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- -[2 5 . A semicpnductor integrated circuit device, 

comprising : 

a memory array having a plurality of word lines, a 
plurality of bit lines, and a plurality of memory cells; 

a processing circuit coupled to said memory array 
via a plurality of signal lines; 

an input /output circuit coupled to one of the 
plurality of signal lines; and 

a switching circuit inserted between the plurality 
of signal lines and said input/output circuit 
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TITLE OF THE INVENTION 
5 Neural Network Processing System Using Semicon- 

ductor Memories 




BACKGROUND OF THE INVENTION 

The present invention relates to a data process- 

10 ing system having a memory packaged therein for real- 
izing a large-scale and fast par^l leKdistributed pro- 
cessing and, more specifically, ltd a neural network 
process i ng. sys tem. 

Jthrtf^paral 1 el distributed data processing using 

15 j><r neural network^ cal 1 ed the "neuro-comput i ng" (as ^ 
will be shortly referred to as the "neural network 
processing") is noted in the field of acoustics, 
speech and image processing, \as\'^descr i bed fiji eithej on ^ 
PP* 145 - 168, "Parallel networks that learn to 

20 pronounce English text. Complex Systems 1 by Sej- 

nowski , T, J., and Rosenberg, C. R. 1987,'^^^ "Neural \/ 
Network Processing" published by Sangyo Tosho and 
edited by Hideki Asou. In gh§ neural network process- 
ing, a number of processing el eraen ts, cal 1 ed ^hel "neu- 

25 rons '*^onnec ted In a networl^^^xchange ^th^ data through 
transfer lines cal 1 ed (\he^ "connect i ons " for high-grade 
data processing. In pach neuron, the data (i.e., the 
outputs of the neurons) sent from another neuron are 
subjected to simple process^g] such as multiplications 

30 or summations. S i nee \th^ process i ng In the individual 

neurons and [th^ process I ng of different neurons can be ^ 
carried out in parallel, the processing 
is advantageous in principle [Ii^J'^lTs^ data process- 

ing. Since algor i thms ^or learnings)^ for setting the 

35 connection weights of the neurons for a desired data 
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processing have been proposed, \th^ data processing can 
be varied for the objects, as described eithe"^^?. «/ 
533 - 536, "Learning representations by back-propaga- 
tion errors.". Nature 323-9 (1986a) by Rumelhart, D. 
E. , Hinton, G. E. and Williams, R. J., ^r i^'*2nd Sec 
tion of "Neural Network Processing" published by San- 
gyo Tosho and edited by Hideki Asou. 

' 

SUMMARY OF THE INVENTION 

First Sai' al 1^, the operating principle of the neu-^ 
ral network will be described in connection with two 
representative kinds: the mul ti -layered network and 
the Hopfield network. Fig. 2(a) shows the structure 
of the multi-layered network, and Fig. 3(a) shows the 
15 structure of the Hopfield network. Both of these net- 
works are constructed ^Lf th^'^nnect i ons of neurons. s/ 
Her^ ^re usedj the terminology (of\ "neurons"j; which will \/ 
be called th^'^'^odes" or "processing el eraen ts^^^gi the ^ 
case may be. The. direct ions of the connectilom arrows vX 
20 indicate'^ |hose of transferring the outputs of neurons} 
In the multi-layered network, as shown in Fig. 2(a), 
the neurons are stacked in multiple layers so that the 
neuron outputs are transmitted in the direction from 
the input to output layers only. Input signals INx, - 
25 - and IN„ are input^d^ to the input layer, and out-v/ 
put signals OUTx , - - - , and OUTn are output\ed^ from 
the output layer. In the Hopfield network, on the 
other hand, the neuron outputs are fed back to an 



identical neuron and are transferred in two^aygHbe- n/ 
30 tween"* arbi trary ^w^ neurons. The feedback to the ^ 

identical neuron may be omitted. 

Figs. 2(b) and Fig. 3(b) show the processing 

principle to be accomplished in the neurons. This 

processing principle is similar in any network and 
35 will be described in connection with the multi-layered 



network with reference to Fig. 2(b). Fig. 2(b) shows 
a j-th neuron in the (S+l)th layer in an enlarged 
scale. This neuron is fed through the connection with 
the output values Vxs. - • Via, - - and V„as of 
5 the neurons in the preceding layer, i.e., the S-th 

layer. Here, letters ns indicate the number of neu- 
rons in the S-th layer. In the neuron, the'^products, 

ViaT^'jt, - - ViaT^ji, and Vna8T**j„a of the 

l^nputtedj output values Via. - - -, Via. - - - , and 

10 Vnaa of the neurons and the connection weightSyT^ji 

and so ony are calculated by means of a multiplier MT. ^ 
Next, the sura of these products and an offset 0 ja-^i 
is calculated by means of an adder ADD. The offset 0 
ja*i may be omitted, as the case may be. Moreover, 

15 the result is input^e^ to a^ircult D' fo^ nonlinear / 
transfer f unc t ron^ ?o"^ain the output ^lue V.a^x of ^ 
the neurons. The nonlinear transfer function circuit 
D has characteristics, as shown in Figs. 2(c) or 2(d), 
and outputs ^n output} g(x) for an input x. Fig. 2(c) \/ 

20 shows an example of the nonlinear transfer function 

\for outputtin^^a binary output gl or g2 (in dependency ^ 
upon whether or not the input x exceeds a predeter- 
mined threshold value xth^^^^ndj Fig. 2(d) shows an ex- \/ 
ample using a sigmoid function for issuing continuous 

25 outputs. The nonlinear transfer function circuit D is 
given other characteristics, if necessaryL As the 
case may be, on the other hand, the circuit Q ma/* be v/ 
given 1 inear characteristics. 

The processing principle described above is also 

30 similar in the Hopfield network, as shown in Fig. 

3(b). In the Hopfield network, however, not only the 
output of the neuron of the'' 1 ayer^ \£recedi ng by one] but ^ 
also the outputs of all neurons are Input^e^ to one s/ 
neuron. In the multi-layered network, as seen from . 

35 Figs. 2(a) and 2(b), one process\i^n^ is ended by^'^feed- ^ 
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Ing the output values of the neurons of the input 
layer ^t f Irs tl and^ by updating the output values of t/ : 
the neurons in the next layer and then b/^h^. output(^^ 
values of the neurons of the ^econdaryj next 1 a ye'^^^^'*^^ 
5 the Hopfield network of Fig. 3(a), on the other hand, 
the output values of the individual neurons can be up- 
dated at suitable tir^jigsj because of^f'ack of any lay- v/ 
er. In this Hopfield network, all the neuron output 
values are suitably given, arfd [their ^"P^^^^^Sl^^on- 

10 tinued til| they come to an equi 1 ibr ium '^^t'et'^'^^ ^ 
ne^lwork^, * i iT^hi ch the output values of all neurons 
are simultaneously updated, is called the "synchroniz- 
ed Hopfield network", whereas the^^twor^ in which 
the^output values are updated at arbitrary timings, 

15 '^r^'^called the "unsynchroni zed Hopfield network"^o ^ 
that they are distinguished^. ^ 
One method used for ^ccompl ishin^^he^^^i^r^^ ^ 
tionecQ neural n^J^Y^o^H^ employed ^e^software ^ 

whereas the othe?^^n§ hardware. According to the 

20 method employing software, the processing of^*^ ^ 

neurons is carried out with a program written in com- 
puter languag€?g] so that the number or structure of the ^ 
neurons can be easily changed. Since, however, the 
processing is sequentially performed, the former 

25 method is disadvantageous In that the data processing 
time is abruptly elongated for an increased number of 
neurons. In the Hopfield network using an n number of 
neurons, ^n n times of products have to be calculated 
for ^pdatin^^^^Ui^e output of one neuron. In order to 

30 update the output values of all neurons at least once, 

therefore, an n* times of products \need t^ be calcu- v/ 
lated. In other words, the number of calculations 
will increase {lff['^the order of n* with (th^'^i ncrease in 
the neuron number n. As a result, the data processing 

35 time will i ncrease Q i^'^he order of n* If Ithe\multi- ^ 
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plications are sequentially accomplished. 

According to the method empl oy I ng %h9^ hardware , ^ 
the processing time can be shortened by ^hangi n^^Ulie ^ 
neurons to be multiplied Into the hardware. Another 
5 trial for speeding up processing has been made by 

executing the processing In parallel with a number of 
hardware neurons. If, however, the number of neurons 
Is enlarged, the number of wiring lines acting as the 
Signal lines between the neurons will increase Qjil''the \/ 

10 order of n*,. thus making It difficult to realize a 
large-scale network. 

The method of solving the wiring problem Is ex- 
emplified on pp. 123 - 129 of Nikkei Mlcrodevlce, 
March, 1989, ^^^111 be described In principle In Fig. \/ 

15 4. Fig. 4 shows an example, In which a multi-layered 

network'v composed of three layers, each having three ^ 
neurons/ Is constructed of analog neuro-processors AN? ^ 
and SRAM^^ The ANP is made by integrating one multi- 
plier tM^'^and one adder ADD of Fig. 2(b) and a non- ^ 

20 linear transfer function circuit D tint^^one chip. 

Another chip^SRAM is stored with the connection weight 
belonging to each neuron. The neurons of different 
layers are connected through one signal line called 
the "analog common bus". Since the neuron output 

25 value of an Input layer is Input^e^l from the outside, 
the ANP and SRAM corresponding to the neurons of the 
input layer are omitted. 

The operations are as follows. First ^f ali, the ^ 
connection weigh for each ANP Unnecessary for a desired ^ 

30 data processing^ Is read In each SRAM from the outside. \/ 
Next, an Input analog signal corresponding to the out- 
put value of one neuron in the Input layer is inpu^ ^ 
te^. The input signals are Input^e^ In parallel dl- ^ 
rectly to the ANPs corresponding to the neurons of the 

35 middle layer. The weight data are read out from the 



SRAM to each ANP in synchronism with the input signal. 
Next, the product of two signals is calculated, and 
the result is stored in each ANP- Next, the input 
analog signals corresponding to the output values of 
5 other neurons of the input layer are input^e^, and 

their product is calculated. The calculation result 
is added to the value which is stored in each ANP of 
the middle layer. After similar calculations have 
been accomplished for the input analog signal cor- 

10 responding to the output value of the last neuron of 

the Input layer, the neuron output values Vxa, Vaa and ' . . ^ 
Vaa Of the middle layer are i£onsecut I vely^ output|t^e^'' to 
the analog common bus of the middle layer by the non- 
linear transfer function circuit in the ANP so that 

15 ^^*Tcalculat ions %Xmi lar the foregoing one^ are continued, 
Finally, the neuron output values Vxa, Vaa and Vaa of 
the output layer are Qpnsecut 1 vel^ output^e^^^to^ ^ 
analog common bus of the output layer by the nonlinear 
transfer function circuit in the ANP of the output 

20 layer. 

According to the example of the prior art de- 
scribed above with reference to Fig. 4, the wiring 
problem can be avoided by driving the common bus in 
the time sharing manner. Since, moreover, the multl- 
25 plications of the number of the neurons in one layer 
can be carried out In parallel, the data processing 
rate can be drastically raised, as compared with the . 



method^gccordlng to the\ sof tware ,^^(§3 a result of the 
speed-up of the processing by the hardwar]^. ^ ^ 

30 Since, however, the ANPs and the SRAMs are"^ formed 

Inl different chips, it is difficult to real I ze'^^he} i/ 
large-scale network Gji high densit^. Even if thirty \/ 
two neurons could be Integrated over a square chip of 
10 mm, as described on pp. 129 of the above-specified 

35 Nikkei Micro Device, March, 1989, one adder, one mul- 
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tiplier and one nonlinear transfer function circuit 

-TO 

need^be prepared for each neuron. Hence, it is dif- \y 
ficult to integrate several hundreds or thousands'^eu- v// 
rons ^ve^^ne chip. ^ 
5 Moreover, the examples of the prior art described 

above H'^'' accompanied by the following problems when \/ 
they are applied to practices. An appl icat i on of the 
multi-layered network is exemplified by^^output^n 
pronunciations and accents of English letters input^ed^ ^ 
10 to a three- layered network, as described on pp. 145 - 
168 of Complex Systems 1 (1987) entitled "Parallel 
networks that learn to pronounce English text" and 
edited by Sejnowski, T. J., and Rosenberg, C. R. Sev- 
en English letters are encoded as the neuron output 
15 values of the first layer, and the codes corresponding 
to the pronunciations and accents of the central one 
of the aforementioned seven letters are output^ed) as 
the output values of the twenty six neurons of the 
third layer. In this example, ^ dependenc'f^^'^up^ the 
inputs, the output values of the neurons of the output 
layer may fail to coincide with the codes of the pre- 
defined pronunciations and accents but may give fuzzy 
values. It is, therefore, necessary, to find^u£^the v 
codes closest to those of all the pronunciations and 
25 accents'^coraparef^ andi'^to make ^hed'^&s^the correct an- ^ 
swer. These comparisons between the output values and 
the desired values (i.e., the codes of the pronuncia- 
tions and accents in the aforementioned example) of 
the outputs are likewise required for ^h^ pattern v/^ 
30 recognition by the neural network. This point is not 
taken into consideration in the aforementioned ex- 
amples of the prior art so that the examples are fin- ^ 
convenient when applied to, the practical problem. ^ 
In the ^f orementione^'examples of the prior art, \/ 
35 moreover, the connection weight necessary for the 
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desired data processing is determined by an external 
computer, and the result is wr i t ten ^n)'* the SRAMs of 
Fig. 4. As a result, the learning is wholly performed 
by (Lh^ software^'^o thaT\ i t UsJ di f f icul t to carry out ^ 
at C^l high speed. 

In order to solve the problems thus far describ- 
ed, according to the present invention, there is pro- 
vided a neural network processing system which com- 
prises: a memory for storing neuron output values , 
connection weights, the desired values of outputs, and 
data necessary for learning; an input/output circuit 
for writing or reading data in or out of said memory; 
an ari thmet ic ci rcui t for performing a processing for 
determining the neuron outputs such as the product, 
sum and nonlinear conversion of the data stored in 
said memory, a comparison of the output value and its 
desired value, and a processing necessary for 
learning; and a control circuit for controlling the 
operations of said memory, said input/output circuit 
and said arithmetic circuit. The arithmetic circuit 
is constructed to include at least one ^f an\ adder, a 
multiplier, a nonlinear transfer function circuit and 
a comparator so that at least a portion of the proces- 
sing necessary for determining the neuron output 
values such as the product or sum may be accomplished 
in parallel. Moreover, these circuits are shared 
among a plurality of neurons and are operated in a 
time sharing manner to determine the plural>^neuron ^ 
output values, S 1 1 1 1/ moreover , the^for^entloned 
comparator compares the neuron output value determined 
and the desired value of the output In parallel. 

Since the adder, the multiplier and the nonlinear ^ 
transfer function circuit are shared among the plural^^ ° 
neurons, as described above, the system for calculat- 
ing the neuron output values In the neural network 
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using [the} numerous neurons can be realized by using a 
small number of circuits. Since, moreover, at least a: 
portion of the neuron process i ngy such as the product ^ 
and sum^ is performed in parallel by the aforementioned y 
arithmetic circuit, the data processing can be 
complished at l§][high speed, ^Ince, still moreove^C \/ 
the output value obtained and the desired value of the 
output can be compared in parallel by the aforemen- 
tioned comparator, the distance (i.e., the similarity 
between the obtained output value and the expected 
value, e.g., th^i^'^^umlning] distance) between the ob- s/ 
tained output value and the desired value can be 
determined at ^he^ high speed. "Since, furthermore, at ^ 
least a portion of the processing necessary for the 
15 learning is accomplished by the hardware of the sys- 
tem, the learning^an be Speeded uq. 

An object of the present invention is to solve 
the problems owned by the aforementioned prior art. 
Another object of the present invention is to 
20 provide a system for carrying out the calculations or"^ 

neuron output values atla^high speed with, a small num- y 
ber of circuits in the network containing a number of 
neurons. 

Still another object of the present invention Is 
25 to give the aforementioned system a function to com- 
pare the neuron output value and the desired value at 
^"^high speed. 

A further object of the present invention is to 
give the aforementioned system a function to process 
30 at least a portion of the processing necessary for the 
learning. 

These and other objects and many of the attendant 
advantages of the present Invention will be readily 
appreciated as the same becomes better understood by 
35 reference to the following detailed description when 



considered in connection with the accompanying draw- 
ings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig, 1 shows one embodiment of the structure, in 
which the data processing system according to the 
present invention is embodied over a single semicon- 
ductor chip; 

Figs. 2(a) and 2(b) are diagrams showing the 
principle of. the mul t i -1 ayered neural network; 

Figs. 2(c) and 2(d) are diagrams showing the ex- 
amples of the characteristics of the nonlinear trans- 
fer function circuit D; 

Figs. 3(a) and 3(b) are diagrams showing the 
principle of the Hopfield neural network; 

Fig. 4 shows an example of the neural network 
processing system using the plural chips according to 
the prior art; 

Figs. 5(a) and 5(b) show embodiments of the 
method for parallel processing of the mul t i - 1 ayered 
neural network and the Hopfield neural network, re- 
spectively; 

Figs. 6(a) and 6(b) show second embodiments of 
the method for parallel processing of the multi- 
layered neural network and the Hopfield neural net- 
work, respectively; 

Fig. 7(a) shows one embodiment of the structure, 
in which the data processing system according to the 
present invention is realized by using a memory array 
capable of reading out a number of data on data lines 
by selecting one word line; 

Fig. 7(b) shows one embodiment of the structure, 
in which the data processing system according to the 
present Invention Is realized by using two memory ar- 
rays capable of reading out a number of data on data 
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lines by selecting one word line; 

Fig. 8(a) shows one embodiment of the correspond- 
ing relations of memory cells to the neuron output 
values and the connection weights, i n^case"^^The multi- 
5 layered neural network is realized by using the em- 
bodiment of Fig. 7(b), that is to say, the embodiment 
in which the neuron output value and the connection 
weight are realized with each memory cell; 

Fig. 8(b) shows one embodiment of the character- 

10 istics of the nonlinear transfer function circuit D 

which Is suitable i n^case'^'^^nary memory cells are^used 
In the embodiment of Fig, 8(a); 

Fig, 8(c) shows one embodiment of the charac- 
teristics of the nonlinear transfer function circuit D 

15 which Is suitable In^-case^ tetral memory cells are used 
in the embodiment of Fig. 8(a); 

Fig. 8(d) shows one embodiment of the method of 
selecting the word lines and the memory cells in the 
arithmetic mode in the embodiment of Fig. 8(a); 

20 Fig. 9(a) shows a second embodiment of the cor- 

responding relations of the memory cells to the neuron 
output values and the connection weights, I n^case^^'^The 
multi-layered neural network is realized by using the 
embodiment of Fig. 7(b), that Is to say, the embodl- 

25 ment in which the neuron output values and the connec- 
tlon weights are realized a plurality of memory 
cells; J 

Fig. 9(b) shows one embodiment of the charac- 
teristics of the nonlinear transfer function circuit D 

'iu which Is suitable l n'' case*^ the neuron output values and 
the connection weights are equlvalently expressed by a 
plurality of memory cells In the embodiment of Fig. 
9(a) ; 

Fig. 9(c) shows one embodiment of a method of 
35 setting the values Xthl, - - and Xthp of Fig. 9(b) 
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for giving the nonlinear transfer function circuit D 
the sigmoid characteristics; 

Fig. 10<a) shows a third embodiment of the cor- 
responding relations of the memory cells to the neuron 
5 output values and the connection weights, in''''case^e 
multi-layered neural network is realized by using the 
embodiment of Fig. 7(b), that is to say, the embodi- 
ment in which the neuron output values and the connec- 
tion weights are realized fe^'^a plurality of memory 
10 cells; 

Fig, 10(b) shows one embodiment of the charac- 
teristics of the nonlinear transfer function circuit D 
which is suitable i n'^case'i^^the neuron output values and 
the connection weii:hts are binarily expressed by a 
15 plurality of memory cells in the embodiment of Fig. 
10(a); 

Fig. 11 shows one embodiment in'^case'^a DRAM cell 
is used as the memory cell of the embodiment of Fig. 
8(a) ; 

2^ Fig. 12 shows one embodiment of the relations be- 

tween the arithmetic modes and the external signals; 

Fig. 13(a) shows one embodiment of the operation 
waveforms in the memory mode of the embodiment of Fig. 

11; 

25 Fig. 13(b) shows one embodiment of the operation 

waveforms in the arithmetic mode of the embodiment of 
Fig. 11; 

Fig. 14(a) shows one embodiment of the nonlinear 
transfer function circuit D which Is suitable for the 
30 embodiment of Fig. 11 or the like; 

Fig. 14(b) shows one embodiment of the charac- 
teristics of the nonlinear transfer function circuit D 
of Fig. 14(a) ; 

Fig. 14(c) shows one embodiment of the Input/out- 
35 put circuit 10 whlch'^sul table for the embodiment of >/ 
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Fig. 11 or the like; 

Fig. 14(d) shows one embodiment of the potential 
relation between the product and sum output line NO 
and the write line lA in^case'^the nonlinear transfer 
function circuit of Fig. 14Ca) and the write circuit^ 
of Fig. 14(c) are used; 

Fig. 14(e) shows one embodiment of the read cir- 
cuit OUT which is suitable for the embodiment of Fig. 
11 or the like; 

Fig. 14.(f) shows one embodiment of the read latch 
circuit OUTLT which is suitable for the embodiment of 
the read circuit OUT of Fig. 14(e); 

Fig. 15 shows a second embodiment of the multi- 
plier MT in Fig. 11; 

Figs. 16(a) and 16(b) show examples of the SRAM 

eel 1 ; 

i*^ig* 16(c) shows one embodiment i n'^ case** the SRAM 
cells of Figs* 16(a) and 16(b) are used as the memory 
cell of Fig. 8(a); 

Fig. 17(a) shows one embodiment of the operation 
waveforms in the memory mode of the embodiment of Fig. 
16(c) ; 

Fig. 17(b) shows one embodiment of the operation 
waveforms in the arithmetic mode of the embodiment of 
Fig. 16(c); . - 

Fig. 18(a) shows one embodiment in^case^the DRAM 
cells are used in the embodiment of Fig. 9(a) or the 
embodiment of Fig. 10(a); 

Fig. 18(b) shows one embodiment of the structure 
of the block BLKl in the embodiment of Fig. 18(a); 

Fig. 18(c) shows one embodiment of the structure 
of the block BLK2 and the nonlinear transfer function 
circuit D in the embodiment of Fig. 18(a); 

Fig. 18(d) shows one embodiment of the structure 
of the individual nonlinear transfer function circuits 
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DSx (X = 1, 2, - - and p) composing the nonlinear 
transfer function circuit DIO in the embodiment of 
Fig. 18(c); 

Fis. 18(e) shows one embodiment of the charac- 
teristics of the individual nonlinear transfer func- 
tion circuit DSx (X = 1, 2, , and p) composing 

the nonlinear transfer function circuit DIO in the em- 
bodiment of Fig. 18(c); 

Fig. ig(a) shows one embodiment of the structure 
of the nonlinear transfer function circuit DIO which 
is suitable in case the neuron output values and the 
connection weights are binarily expressed in a plu- 
rality of memory cells in the embodiment of Fig. 
18(a) ; 

Fig. 19(b) shows one embodiment of the charac- 
teristics of the nonlinear transfer function circuit 
DSx (X = 1, 2, - - and 2) in the embodiment of Fig. 
18(a); 

Fig. 19(c) shows one embodiment of the charac- 
teristics in the embodiment of Fig. ig(a); 

Fig, 19(d) shows one embodiment of the structure 
of the encoder in the embodiment of Fig. 19(a); 

Fig. 20(a) shows one embodiment of the corres- 
ponding relations of memory cells to the neuron output 
values and the connection weights, in case the un- 
synchronized Hopfield neural network is realized by 
using the embodiment of Fig. 7(b), that is to say, the 
embodiment in which the neuron output value and the 
connection weight are real i zed (wi tl^** each memory cell; 

Fig. 20(b) shows one embodiment of the corres- 
ponding relations of memory cells to the neuron output 
values and the connection weights, in case the syn- 
chronized Hopfield neural network is realized by using 
the embodiment of Fig. 7(b), that is to say, the em- 
bodiment In which the neuron output value and the con- 
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nection weight are real i zed ]wi tj|'*each memory cell; 

Fig. 21(a) shows one embodiment of the corres- 
ponding relations of memory cells to the neuron output 
values and the connection weights, in^case^he unsyn- 
chronized Hopfield neural network Is realized by using 
the embodiment of Fig. 7(b), that Is to say, the em- 
bodiment In which the neuron output value and the con- 
nection weight are realized with a plurality of memory 
eel 1 s ; 

Fig. 21.(b) shows one embodiment of the corres- 
ponding relations of memory cells to the neuron output 
values and the connection weights, ln^case*^^^e syn- a/ 
chronlzed Hopfield neural network Is realized by using 
the embodiment of Fig. 7(b), that Is to say, the em-^ 
15 bodlment In which the neuron output value and the con- 
nection weight are realized with a plurality of memory 
cells; 

Fig. 22 shows one embodiment 1 n** case " the neuron ^ 
output values and the connection weights are enabled 
20 to take positive and negative values by using coding 
bits; 

Fig. 23 shows one embodiment in case'' the system v/ 
according to the present invention Is given a function 
to compare the neuron output values and the desired 
25 values; 

Fig. 24 shows one embodiment of the comparator 
for comparing the data read out to a plurality of data 
line pairs of the memory cell array TG and the memory 
cell array A to calculate the extent of similarity of 
30 the data; and 

Fig, 25 shows one embodiment In which (the updat- ^ 
Ing of] the neuron output values'^Jls speeded uj^ by pro- ^ 
vldlng a register. 
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Fig. 1 shows one embodiment In^case'the data 
processing system according to the present invention 
Is Integrate^^ver a semiconductor chip. 

thete arii'^lntegrated over a semiconductor chip 
(CHIP): a memory (A) for storing data; an Input/output 
circuit (I/O) for performing at least one of the writ- 
ing operation and reading the data in and from said 
memory; an arithmetic circuit (ARTM) for performing 
the arithmetic for determining neuron output values, 
the comparison (I.e., the similarity of the obtained 
output values and the desired values, e.g., the cal- 
culation of the ^ummin^^^dTlT^ance) of the output values 
and the desired values or the arithmetic necessary for 
tlhe) learning by using the data stored In said memory; 
and a control circuit (CNT) for controlling the opera- 
tions of said memory, said input/output circuit and 
said arithmetic circuit. Each of"^luses (BUSl , BUS2, 
BUS3, BUS4, etc.) connecting the Individual blocks is 
made of not only one wiring line but also a necessary 
number of wiring lines. The aforementioned memory can 
be stored with the connection weights and the neuron 
output values necessary for ^he\ neural network proces- 
sing, the desired values of the outputs or the data 
necessary for the learning. According to the present 
embodiment, (th*^ nonlinear network processing such as 
the calculations of ^^e iveijron output values, the com- 
parisons of the (^eslr^dfValues ^^*%e desired values, 
or the calculations necessary for ^hS^ learning can be 
performed In the f o 1 1 owl n2*\manner^ • 

First "^f al^, the method of calculating the neu- 
ron output values will be describedOn the following. 
\Atl'^f IrstV the connection weights necessary for the 
calculations for ^e\ neural network processing, and ^ 
the neuron output values, or the offsets are read out 
In parallel from the memory to the arithmetic circuit 
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through a bus 1. Next, the ^ri thmet i ci\'^such as the 
product and sum or the nonlinear transfer necessary 
for determining the neuron output values are accom- 
plished by the arithmetic circuit, and the obtained 
results are written In the memory through the input/ 
output clrcult^^The operations described above are 
continued ^y ^"necessary number of times to determine 
the neuron output values. The arithmetic circuit may 
either determine one of^pl ural'^'^neuron output values by v/ 
a single operation or perform a portion of calcula- 
tions for determining the neuron output values. Thus, 
the data processing can be accomplished by the various 
-n-etworks such - as-the mul t i - l ayered ne twork or the 
synchronized or unsynchron i zed Hopfield network. In- 
cidentally, In order to update the output values of 
all the neurons synchronously, the synchronized Hop- 
field network needs to be stored with the output 
values of all the neurons ^l ij^^^he^nd of updating th^ 
output values of all the neurons\""^1r?*' thl s case, the 
output values of all the previous neurons may be 
stored In the memory so that they may be used for up- 
dating the output values of the neurons. 

According to the present embodiment, a desired 
number of multipliers, adders and nonlinear transfer 
function circuits necessary for calculating the neuron 
output values may be provided In the arithmetic cir- 
cuit so that they may be-'repeatedly fesedj. This makes vX^ 
It possible to make the number of circuits far smaller 
than \that of] the case In which those circuits are v/ 
prepared for each of the neurons. The example of the 
prior art of Fig. 4 ^s^ requl re^^o prepare] two hund- 
reds multipliers, adders and nonlinear transfer func- 
tion circuits for realizing the mul t 1 -1 ayered network 
having three layers each composed of one hundred neu- 
rons. In the present embodiment, on the contrary, it 



$ t 



- 18 - 



is sufficient to prepare at least one multiplier, at 
least one adder and at least one nonlinear transfer 
function circuit. Even if the multiplications neces- 
sary for updating ^J^neuron output value Qo speed up 
the operations shoul^^be accomplished in parallel, it v/ 
would be sufficient to prepare one hundred multi- 
pliers, one adder and one nonlinear transfer function 
circuit. According to the present embodiment, there- 
fore, the number of circuits can be drastically re- 
duced, as compared with that of the prior art. In- 
cidentally, the above-specified ^ifferenf^ttit become ^ 
(thejlarger for Ih'S, larger scale ^f thel networl?. Slmi- ^ 
lar situations will apply to another network such as 
the Hopfield network. 

Not only the ^r i thme t i c^spted such as multiplica- 
tiongljbut also the amounto£^r i thme t i c^^tbT'^^^ v/ 
out in parallel makes yi^cS'ntrlbution to Ch^ data ^ 
processing speed tin case^^^the neuron output values are 
to be determined. In the Hopfield network using \an] n \/ 
number of nejirons, for example, Qhi^produc ts lof n* ^ 
times have tB^be calculated for updating the output 
values of J^H the neurons, as has been described 
Chereinbefor|r If the mul t Ipl i cat i ons'^arll^^uen t i al 1 y ^ 
laccompl ished}, therefore, the updating of the output 
values of all the neurons takes at least a time period 
of n* times as long as that required for one multi- 
Plication. As a resul^t, the time period required for 
the mu^Upl ications will abruptly i ncrease ^n^^the or- 
der of'square of the neuron number with ^i'^ncrease \X 
in the number of neurons. Similar circumstances will 
also apply to the multi-layered network. This makes 
it desirable to calculate ^he- numerou'^'^mul tipl ications 
in parallel, [here will be described in the f ol lowin^'?^-' 
an example of the arithmetic system for raising the 
data processing speed by making the multiplications in 



t $ 

W - 19 - ^ 



parallel so as to determine the neuron output values ^ 
in the embodiment of Fig. ^'^^^^'-^-'^M 

Fig. S illustrates the mul t i - 1 ayered network at 
(a) and the Hopfield network at (b) of one system for 
5 ^he) paral lel^^^^L^^ In the present embodiment, ^/ 

the products necessary for determining one neuron out- 
put value are calculated, as shown. Specifically, the 
output value of the neuron of the preceding layer, 
which is inputted} to one neuron, and the connection 

10 weight for^ said' output value of the neuron under con- v/ 
sideration are read out in parallel from the memory, 
.and their products are calculated in parallel. Thus, 
. "the" time period required f or the mul t i p 1 i cat i ons will 
increase jli^'^he order of the neuron number with^t^hej/^ 

15 increase in the neuron number. As a result, the data 
processing time can be drastically shortened, as com- 
pared with the case in which the multiplications are 
' ^ sequent ial ly ^ccompl Ishe^. In Fig. 5, only the multi-^ 
plications necessary for updating the output value of 

20 one neuron are'^taccompl ishe^ In parallel. However, the ^ 
embodiment of Fig. 1 should not be limited thereto but 
may naturally add the arithmetic circuits within a 
range allowed by the degree of Integration, to update 
the output values of the plurar^neurons in parallel. 

25 In this case, Ithel data processing can be accomplished 
at \al higher speed. In addition, the parallel (§.rlth- 
metlcl'^can also be accomplished by another system, as ^ 
shown In Figs. 6(a) and 6(b). 

Fig. 6 shows one embodiment. In which the multl- 

30 plications are executed In parallel for a plurality of 
neurons to be fed with the output value of one neuron 
In the multi-layered network of Fig. 6(a) and In the 
Hopfield network of Fig. 6(b). In this method, the 
neural output values and the connection weights are 

35 read out from the memory, and the calculations neces- 
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sary for updating the neuron outpu^ values are ex- 
ecuted bit by bit for the plural«^neurons . This makes 
it impossible to realize the unsynchroni zed Hopfield 
network. Since, however, the time period required for / 
5 the multiplications will increase ^n^^he order of the ^ 
neuron number with the increase in the neuron number 
like the system of Fig. 5, the data processing time 
can be drastically shortened, as compared with the / 
case in which the multiplications are \iequent i al lyj 
10 carried ouItT*^^ 

In the example, of the prior art of Fig. 4, too, 
the ^rl thine tlc^-^are executed in parallel. As will be ^ 
descrlbed^^ the following, however, the structure of ^ 
Fig. 1 can be realized with a smaller number of cir- 
15 cults than that of the exampl e {og'^he prior art. In v/ 
the systems shown in Figs. 6(a) and 6(b), only one 
mulJjjLlier operates in parallel in each neuron, as 
hatch e^i In the embodiment of Fig. 1, therefore, the ^ 
arithmetic circuit may be provided wi th \the\ mul t i - 
20 pliers in a number eaual to that of the neurons to be 

calculated at one time/so that this system can be i/ 
realized with a smaller number of circuits than that 
^H^he case of the prior art in which ^11 thejmulti- 
pliers are provided for all the individual neurons. 
25 In the multi-layered network having three layers/ each ^ 

composed of three neurons, for example, a similar par- 
^^^^^^^^^^^^^^ 1 e can be realized by us i ng ]i_ndi vidual 1]^ three ^ 

^multipliers, adders and nonlinear transfer function ^ 
circui ts[7 for exampl eTl according to the embodiment of 
Fig. lQ«^n th^'^contrary to the case o J^^,^he^rl or art \y 
which is equipped wl th individual 1^ slx^ mul tipl lers, ^ 
adders and nonlinear transfer function circuits. 

Thus, according to the embodiment shown In Fig. 
1, a system for (th^ data processing similar to that of \/ 
35 (th^ neural ne tworl^ us I ng numerous neurons can be real- \/ 



ized with the'^ necessary ^ nimur^ cl rcui ts by sharing ^ 
the adders, multipliers and nonlinear transfer func- 
tion circuits of the arithmetic circuit among thq 
plurar neurons. By executing the ^rl thmet I c'S^^^such as 
the products or sums with the aforementioned arith- 
metic circuit, moreover, the data processing can be 
accomplished at^Thlgh speed. Incidentally, with . 
paraltVl^^*^i^hme 1 1 c^ the number of wiring lines be- \/ 
tween the memory and the arithmetic circuit has to be 
Increased to send many data at once to the arithmetic 
circuit. In Fig, 1, however, the memories and the 
arithmetic devices are arranged over ^^^ommon chip ^ 
so IthatJ the number of ^h^ wiring lines ^^^the bus can ^ 
be easily Increased. 

Although the method of calculating the neuron 
output values has been described hereinbefore, a neu- 
ron output value and its desired value can be compared 
according to the embodiment of Fig. 1. For this com- 
parison, the desired value may be stored in advance In 
the memory so that Its distance from the output value 
obtained by the aforementioned method may be calcu- 
lated by the arithmetic circuit. This operation is to 
calculate the similarity between the desired value and 
the calculated value. At this time, the desired value . 
and the output value^ composed of numerous bitsycan be ^ 
simultaneously read out to the arithmetic circuit and 
processed In paral 1 el yWi th ease by increasing the nura- 
ber of wiring lines ^^''the bus 1. Thus, according to ^ 
the embodiment of Fig. 1. (th^ data processing such as 
^h"^ pattern recognition can be executed atCa"^high ^ 
speed, as compared with the case In which the com- 
parison is accomplished serially bit by bit by using 
an external computer. 

According to the embodiment of Fig. 1, moreover, 
the learning can be accomplished at^'^hlgher speed k/ 
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than that of the case us i ns ^he} so f tware , by executing 

the jar I thme 1 1 c^ne^ for ^h^ learning with the 

ar i thme t^^i^^^i rcui t , This specific embodiment will be / 
describeci^^ereinaf te^. ^ 
5 ^he ^eural network processing is advantageous in y 

that it can process various data by changing the con- 
nection weights. This advantage can be easily ex-- 
plotted according to the embodiment of Fig. 1 by re- 
writing the connection weight stored in the memory. 

10 Moreover, several kinds of connection weights neces- 
sary for different data processings can be stored In 
advance by making the capacity of the memory larger 
than that necessary for calculating the neuron output 
values. In this case, ^here can be attained a merlK ^ 
>^that different kinds of data can be continuously 
processed without losing the time period for rewriting 
the connection wel gh ts'^.^^^'^Tif add I 1 1 on , case'^^umerarj 
Input data are to be continuously processed, the nec- 
essary Input data or the obtained data can be stored 

20 In advance In a portion of the memory. Thus, the 

frequency for switching the reading, calculating and 
output^ ng| modes can be reduced to shorten the pro- 
cessing time, as compared with the case in which the 
operations of reading each Input data In the memory . 

25 and calculating and outputtlng ir^\lo Jhe outside of ^ 
the system] are repeated. 

Next, a more sped fic embodiment^ based upon the ^ 
embodiment of Fig. 1/wlll be described Cln ^^^^J^^^^^" 
In^. For slmpl Icl ty (of descriptions!, the case'^fof 

•^0 glvlngjthe arithmetic clrculf^the function of cal- ^ 
culatlng the neuron output values will be described at 
first, and the method of giving the comparing or ^ 
learning function will be descrlbed^^ lasft. ^ / 

Fig. 7(a) shows one embodiment In^case^a lattice- ^ 

35 shaped memory cell array Is used In the memory of the 
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embodiment of Fig. 1. In Fig. 7(a), letter A desig- 
nates a memory cell array which is composed of: a 
plurality of data lines (D) ; a plurality of word lines 
(W) arrayed to intersect the data lines <D) ; and memo- 
5 ry cells CMC) arrayed at the desired intersections. 

As a result, the signals of the di.fferent memory cells / 
can be read out onto the pluralj'HiVta lines by select- ^ 
Ins one of the word lines. Numeral 12 designates an 
arithmetic circuit (ARTM) • Numerals 10, 11, 13, 14, 

10 15 and 16 designate circuits corresponding to the con- 
trol circuit (CNT) of Fig. 1. The numerals 10 and 15 
designate address buffers for X-addresses and Y-ad- 
dresses, and the numerals 11 and 14 designate a de- 
coder and a driver for X-addresses and Y-addresses, / 

15 respectively. jTheJ^umeral 13 designates an array con- 
trol circuit for controlling the memory cell array. 
Numeral 16 designates a clock generator for generating 
clocks to con tro l[s] the operations of the memories on 
the basis of the signals inputted] from the outside. 

20 Le-tters OUT and WR designate a read circuit and write 
circuit, respectively. A chip select "CS is a chip 
selection signal. A write control signal WE is a sig- 
nal for switching the write and read operations for 
establishing the write operation at a low level and / 

25 the read operation at a high level. Let te^ "RF desig- ^ 
nate an arithmetic circuit control signal for starting 
the arithmetic circuit at .a low level and interrupting 
the same at a high level to act as an ordinary memory. 
In the following, the state of the signal TJE at the 

30 high level will be called the "memory mode", and the 
state at the low level will be called the "arithmetic 
mode". In the memory mode, a desired memory cell is 
selected according to the X-address and the Y-address 
30 that a write data DI can be written in that cell or 

35 so that^^data can be read out from the same cell and 
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outputlt,e^ as a read data DO, In the arithmetic mode, v/^ 
the data stored in the memory cell is read out to the 
arithmetic circuit 12 so that the arithmetic result of 
the arithmetic circuit 12 or the data according to the 
5 arithmetic result can be written in the memory cell 
through the input circuit. By selecting one word 
line, according to the present embodiment, the data of 
all the memory cells on the selected word are out- 
put^ed\ to the data lines. As a result, numerous data v 

10 can be easily latched in the arithmetic circuit 12 so 

that many Ig^ri thmetic^'' can be accomplished in parallel.^ 
In order to calculate the neuron output values accord- 
ing to the present embodiment, the mode^s^hanged at \/ j 
firstj^rjto the memory mode to stop the arithmetic cir- v 

15 cuit, and the necessary connection weight, neuron out- 
put value (i.e,. the input signal at first), offset 
and so on are written in the memory. Next, the mode 
is \change^*^o the arithmetic mode to start the arith- ^ 
metic circuit {t.^ reacf^the necessary data^s rea^ out 

20 to the arithmetic circuit by selecting one word line. 

Next, the result is written in the memory circuit. If 



the read of the data necessary for the \ar i thme t i c^^^^^aTid ^ ^ 
the write of the result are further cont i nuedQ):^^ ^ 
^necessary number of times, ^h^ neural network process- ^ 
25 ing can be accomplished at talhigh speed. As has been 
described above, according to the embodiment shown in 
Fig. 7(a), many data can be written at once in the 
arithmetic circui tU^J^jThe embodiment is suited for ^theA 
paral lei \ari thmetic^TOf the type shown In Fig. 5 or ^ 
30 Fig. 6. Thus, according to the present embodiment, 

paral lel^^ar^ make it possible to execute \y 

^e^neural network processing at^^^^^*^ speed. By 
using the arithmetic circuit 12 repeatedly, moreover, 
the plural^ 'neurons can share the output value* cal- ^ 
35 culatlng circuit to^effect a high integration (easi 1^]. ^ 
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In^case. on the other hand, '*paral leK^ri thme tics] ^ 
are to be accomplished by using the data stored in the ■ 
memory cells on the p 1 ural^^^vH^rd lines 

primary storage\can be provided in the arithmetic cir- 
5 cui t so that it may Tonce^ s tor e/ the data^ ob tai ned by v/ 
selecting the word lines and execute th^^rl thmetlci) \/ 
of the stored data together with the data read out by 
selecting other word lines. 

As In the embodiment shown In Flg» 7(b), more- 
10 over, two memorles/A and B can be provided. In Fig. \/ 
7(b), characters 13A and 13B designate array control 
circuits for controlling the memory cell arrays A and 
B, respectively. Other circuits such as a decoder Is 
not shown in Fig. 7(b). According to the structure of 
15 Fig. 7(b), the "^ata of the\ memory cell'lf^n the two ^ 
word lines of ^he] memory cell arrays A and B can be ^ 
written in the arithmetic circuit by-selecting one 
word line each of the memory cell arrays A and B. 
By using the structure of Fig. 7(b)'^^tRe memory arrays^ l/^ 
can be«*^separately (^sed\ according to the kinds of data 
such that ^he\ memory cell array A ^1 gf^tore^] ^l th) the 
neuron output value whereas [th^ memory cell array B^sjy 
stored wlthj the connection weight, 



the controls of 



rea ding or writing operat I ons ^Xn be s I mp 1 1 f I ejdlT" In- 




25 cldentally, In the embodiments of Figs, 7(a) and 7(b), 
the write data DI and the read data DO may be process- 
ed In plurarlty and In parallel, or "^e^ array^ A and B ^ 
may b^^^'separately litrovldedj wl th the read circuit OUT 



and the write circuit WR. 

30 In the embodiments of Figs. 7(a) and 7(b), the 

selection of a specific memory cell can be accom- 
plished like the ordinary memory according to the 
address. By changing the order of selecting the 
address, therefore, those embodiments can be flexibly 

35 applied to the various networks or various parallel 



arithmetic systems. 

In the embodiments of Figs. 7<a) and 7(b), the 
memory can be exemplified by a highly Integrated semi- 
conductor memory such as the DRAM or SRAM. In this 
5 case, the memory can store many data so that a large- 
scale network can be integrated Into one chip. 

Next, the method of realizing the multi-layered 
network by using the structure of Fig. 7(b) will be 
described^^i*^n detail the following. The parallel 
10 arithmetic system is exemplified by taking the system 
of Fig. 5(a). It is assumed that the number of layers 
be ro and that the nxlmber^ of neurons in each layer be 
n. Incidentally, the offset 0 of each neuron, as r 
shown In Flgfej. 2(b) or Fig. 3(b), will be omitted here \/ 
15 ^o a^ to simpl i fy the description. As is apparent v/ 
from Fig. 2(b) or Fig. 3 (b) , however , the of fset of 
each neuron can be handled like the output from an- 
Other ordinary ^neuron.'^l^l th^er by provldirf^'^one neuron ^ 
having an output value of 1 at all times to set^ing\ ^ 
20 the connection weight of It and each neuron [ayHhe i/ ^ 
Offset 0 •i6r by 1 ncreas I ngjHhe neuron output value, v 
which Is to l^e lnput|ted/ l.oP^'^each neuron, by l*" to set v 
the value fa tl'''' the offset 0 of each neuron and the cor-/ 
responding connection weight ^t^l so that their pro- ^ 
25 duct may be added to the total sum of the products of 
other neuron output values and the connection weights. 
Fig. 8(a) shows one embodiment In which the memory 
eel Is |are^ correspond to the connection weights and the 
neuron output values. Letter D designates the non- 
30 linear transfer function circuit; characters cl, c2, - 
- -, and cn designate the adders; and characters ml, 
ra2, - - and mn designate the multipliers. The ad- 
ders cl, c2, - - cn'^^cons ti tute ^1 together^ the ^ 
multi-Input adder ADD of Fig. 2(b). The memory cell 
35 array A Is stored with the neuron output values, and 
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the memory cell array B is stored with the connection 
weights. Although what is shown In Fig,;-8(a) is the 
memory cells for storing the neuron output values and 
the connection weights^ it is quite natural that the 
5 memory cells ^'^"be stored with other data such as the 
offsets 0 of the individual neurons or the data nec- 
essary for the learning ^ay be provided), if necessary. ^ 
As shown, the memory cells located at the intersec- 
tions of the word lines s and the data lines i in the 

10 memory cell array A are stored with neuron output 

values Via* In other words, the output values of the 
neurons of the common layer are arranged on the common 
word line. In the memory cell array B, the memory 
cells located at the intersections between the word 

15 lines Cs, j) and the data lines i are stored with con- 
nection weights T^ij. / 

/Figs* 8(b) and SCcr^show one embodiment of the 
input/output characteristics of the nonlinear transfer 
function circuit D. Fig. 8(b) shows the embodiment 

20 having binary outputs gl and g2 . Characters xl and x2 
indicate the lower limit and the upper limit of the 
input X, respectively. In Fig. 8(b), the output is 
g2, if the input x exceeds the threshold value xth, 
^VuV^gl\if no5- Therefore, the embodiment of Fig. 8(b) ^ 

25 is suitable when the memory cell used is binary. Fig. 
8(c) shows an embodiment having tetral outputs ga and 
gb between the outputs gl and g2 . The present embodi- 
ment is a suitable example when the tetral memory 
cells are used. The gap between the elements gl , ga, 

30 and g2 can naturally be changed, if necessary, al- 

though they are shown equally distant in Fig. 8(c). 

case'^the memory cells are exemplified by those ^ 
capable of storing data having continuous values, 
i.e., the so-called "analog values", the nonlinear 

35 transfer function circuit D to be used may have the 



characteristics shown in Fig. 2(d). 

'\_ Fig. 8(d) jShowe^ one embodiment of the correspon- 
dences in the embodiment of Fig. 8(a) between the word 
line selecting method for determining the neuron out- 
put values of the final layer from the neuron output 
values of the input layer and the write destination 
addresses. The operations of Fig. 8(a) will be de- 
scribed in the following with reference to Fig. 8(d). 
The neuron output values Vxx, Van - - and Vnx of 
the Input layer are written In advance in the input/ 
output circuit (although omitted from Fig. 8) in the 
memory cel ls on the word 1 ine \bi^ S =1 of Sjb^ array A. 
First ^f all\, the word lines of s \= 1 of ^eV^array A \/ 
and the word lines of (s, j) = (1, 1) of \the^ array B ^ 
are^sei^^x^d simultaneously, although not necessarily y 
"complete^^ the neuron output values Vxi, V^i , - 

md V„x of the Input layer are output^ed^ to the v/ 
data lines of i = 1, 2, - - and n of array A. ^ 

On the other hand, the connection weights T^n, T^ia, 
- - and T^in are output^edi to the data lines of i = ^ 
1,2,---, and n of array B. These values are 

lnput^ed}ito the multipliers ml, m2 , - - and mn so 
that their products T^xxVxx, T^xaVax, - - and 
T^xnVnx are lnput\e^ to the adders cl, c2 , - - and 
cn. The results (T^xxVxx + T^xaVax, - - + T^xnV„i) 
are lnput!{_ed^ to the nonlinear transfer function clr- ^ 
cult D. The output of this nonlinear transfer func- 
tion circuit D is wiritten through the write circuit WR 
(although omitted) In the memory cells corresponding 
to the write destination addresses of (s, I) = (2, 1) . 
In^e^ array A. Thus, the value of the first neuron w 
output value Vax of the second layer is calculated- 
Next, the word line of s = 1 of ^e) array A and the ^ / 
word line of (s, j) = (1, 2) of ^ej array B are simul- v 
taneously selected. Then, the neuron output values 



Vii. Vai, - - -» and V„x of the Input layer are out- 
put^ed^ data lines of i = 1, 2,. - - and n of ^ 

^e^array A. On the other hand, the connection <^ 
weights T^ai. T^aa, - - and T^2„ are output^e^ to ^ 

5 the data lines of 1 = 1,2, and n of array ^ 

B. These values are input^^ to the multipliers ml, ^ 
m2, - - and ran so that their products T^axVxx, 

T^aaVax. and T^a«V„x are input^e^ to the adders J 

cl, c2, - - and cn. The results CT^axVxx + 

10 T^aaVax, - " + T*anV„x) are inputted! to the non- ^ 
linear transfer function circuit D. The output of 
this nonlinear transfer function circuit Is written 
through the write circuit (although not shown) in the 
memory cells corresponding to the write destination 

15 addresses of (s, I) = (2, 2) In array A- Thus, 

the value of the second neuron output value Vaa of the 
second layer Is calculated. All the neuron output 
values can be calculated by continuing the operations 
thus far described according to Fig. 8(d). According 

20 to the present embodiment, one neuron output value can 

be determined by executing the read^n^ and wrlt^ng\ yy^ 
operations once In the arithmetic mode so that the , 
neural network processing can be accomplished at ^al v/ 
high speed. {Si nce^ l^oreover ,'**Hhe arithmetic circuit ^ 

25 can be shared among all the neurons, high Integra- v/ 
tlon can be'*Qpad^. Incidentally, Fig. 8(d) shows'^one 
example of ^ ass Ignmen t of the memory cells, and the ^ 
present Invention should not be limited thereto but 
^an^p^e modified In various manners. For example, the v/ 

30 plura/^Tnput data gai^'^e continuously proces^^, as ^ 
has been descr I bed \herel nbef ore/. In this case, a ^ 
plurality of sets of neuron output values of an Input 
layer are required. For this operation, the neuron 
output values of the Input layer corresponding to the 

35 plurar Input data may be written In advance on the v/ 
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plurar^di f f erent word lines of ^be^ array A so that 
they may be^^consecutively |use<^ Thus, the neuron out- i/ 
put values of the Input layer need not be read In for 
each data processing so that the data processings can 
5 be continuously^accomplished at \thej high' speed\ 

[Here is useq one memory eel 1^ for storing the ^ 
neuron output value and the connection weight. This 
allows only binary values to be taken as the neuron 
output values and the connection weights i n" case'* the ^ 

10 binary memory cells are used. By using the mixlti- 
valued memory cells, as Ul^s beeri^ described |herei n- 
befor^, the neuron output values and the connection 
weight values could be increased, but the multi-valued /^,^+^ 
memory cells may have ^ tq'' rel iabi L l ty ^ade deficient] y 

15 Ifey thj problem lo| S/N ratio. In this case, a pluraii- 

ty of memory cells ^ca^^lse used for storing the neuron \/ 
output values and the connection weights, as will be 
descr ibed^*^n the following. ^ 
Fig. 9(a) shows one embodiment of the case in 

20 which a* p ^umbe^l of memory cells are used for storing i/ 
one neuron output value and in which a'^q SumbeQ of \/ 
memory cells are used for storing one connection 
weight. The suffix i , j or s appearing in Fig. 9(a) 
to Indicate the neuron output value or the connection 

25 weight corresponds to that of the embodiment shown in 

Fig. 8. In the embodiment of Fig. 9(a), the'^p ^umber] v^. 
of continuous memory cells on one word line in ar- ^ 

tblv a express^one neuron output value, and the'^'^ (gum- 
be?^ of continuous memory cells on one word line in the 

30 array B express^one connect ion weight . ^ 
The calculations of the neuron output values are 
carried out in the following manner. First ^f al^, 
like the embodiment of:Flg. 8, the word line of s = 1 
of the array A and the word line of (s. j) = (1, 1) of 

35 the array B are'^slraul taneously \selectedj. Then, to the ^ 
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data line group of i = 1, 2, - - and n composed of 
• p number data lines of^ array A, there ^r^^out- ^ 
put^e^ the data expressing the neuron output values of 
Vir. Va», - - and V ni of the Input l*ayer» which are 
Input^tedJ group by group, to the adders al^ a2, - 
and an. aQTo the d^Ata i:^lne group of^i = 1» 2, - - ' v/ 

and n composed of q number of data' lines of the array ^ ^ 
B, on the other hand, there are output ted the data ex- ^ 



pressing the connect ion weigh ts T^xi, T^xa. - - and ^ 
10 T^in, which are inputted group by group to the adders >/ 
bl, b2, - - and b3j./ gy the Aforementioned adders y 
al , a2 , - - - , and an , and bl , b2 , - ^ - , and bn 3 l^he ^ 
neuron output values Vxi, Vax, - - and V„i and the 
connection weights T^n, T^ia, - - -t and T^m are 
15 composed'* and Input^ec^, as shown, to the mul tlpl iers >/ 
ml, m2, - - and mn to produce the products T^^'iVxx, 
T^xaVax. - - and T^xnV„x. These products are In- 
put^e^ to the adders cl, c2, - - and cn so that v/ 
their results (T^xxVxx + T^xaV^x, - - + T^xnV„x) 
20 are lnput]te^ to the nonlinear transfer function cir- \/ 
cui t D. The output of the nonlinear transfer function 
circuit is written through the write circuit WR (al- 
though not shown) to the p number of memory cell 
groups corresponding to the write destination address 
25 (s, i) = (2, 1) In (Th^ array A* The output values of v/^ 
all the neurons can be determined by continuing simi- 
lar operations by using the same address as that of 
Fig, 8(d). ' 

Since, according to the aforementioned embodi- 
30 ment, one neuron output value Is expressed with [the] p ^ 
number of continuous memory cells on one word line In 
(the] array A, the multi-valued neuron output values can 
be expressed by using the binary memory cells. Since, 
moreover, one connection weight Is expressed wl th ^hej ^ 
35 q number of continuous memory cells on one word line 



- 32 - 



in^e^array B, the mul t i -val ued connection weights ^ 
can be expressed by ^^^^S^}}}^^\h}tij^^^ memory cells. As 
a result, the (inul 1 1 -value^'^such as the neuron output ^ 
values or the connection wei^:htS; can be expressed^*"^! t^v/ 

5 the binary memory cells* In the aforementioned em- 
bodiment, moreover, the frequency of switching the ad- 
dresses is identical to that of the embodiment of Fig- 
8 so that the data can be proceissed at \a]higt^^ speed ^ 
like^^the embodiment of Fig, 8, In order to write the v/ 

10 result of the nonlinear transfer function circuit in 

\ih§P number of memory cells expressing the neuron ^ 
output values, ^e\p number of writing operations may ^ 
be continuously executed but can be easily accomplish- 
ed in parallel by providing the p number of write cir- 

15 cuits- In this case, it is possible to avoid the time 
losses for writing^the plurar memory cells. By ^o- v/ 
viding a plurality of read circuits, on the other 
hand, it is naturally possible to increase the speed , 
Of the reading operations. Incidentally, if a^ulti-i' 

20 input circuit is used fe^^^the multipliers ml, - - ^ 
and mn in the embodiment of Fig. 9(a), a similar cir- 
cuit can be realized without providing the adders al , 
- - -, and an, and bl, - - -, and bn. In addition, 
the structure of the ar i thmet Ic circui t can be modl- 

25 fled In various manners. 

The embodiment of Fig. 9(a) uses |_hej p number of >/ 
memory cells for storing the neuron output values and 
^he^ q number of membry pells for storing the connec- 
tion weights. In other words, the neuron output 

30 values are expressed in ^he\ p bits, and the connection s/ 
weights are expressed in ^h^ q bits. Since there are 
various methods of expressing the data in'^^hej plural' v 
bits, an expression method may be selected. If neces- 
sary, and It Is accordingly sufficient to design the 

35 characteristics of the adders, the multipliers and the 



nonlinear transfer function clrcui ts . For example, 
the neuron output values can be expressed wi th the 
number of \such: ones of th^ memory eel 1 s ; of p bi ts ex- \/ 
pressing the neuron output values ^^"Kave a content of n/ 
5 1. Fig. 9(b) shows one: embodiment of the input/output 
characteristics of the nonlinear transfer function 
circuit D suited for suchN:ase* In Fig. 9(b), charac- v/ 
ters gl, g2, - - and gp designate the p number of 
outputs of the nonlinear transfer function circuit D. 

10 Their individual outputs take the value 0 or 1, which 
is written in the p number of corresponding memory 
cells through the write circuit WR (although not 
shown). The outputs gl , g2 , - - and gp taKe the 
value 1, when the individual inputs exceed the values 

IS xthl, xth2, - - and xthp, and otherwise take the 

value 0. These values xthl, xth2, - - and xthp may 
be at an equal or arbitrary distance between the upper 
limit xl and the lower limit x2 of the inputs. If the 
distance between the values xthk and xthk+1 (k = 1, - 

20 - and p-1) is set, as shown in Fig. 9(c), the non- 
linear transfer function circuit g can be^^^eal i ze41 to V 
have ^he^ sigmoid characteristics. According to the ^ 
present embodiment, one neuron output value can be 
given P number of values by the p number of memory ^ 

25 cells. In the present embodiment, the p number of 

memory cells expressing the neuron^output values are 
equivalently handled. Speci i i calmly the Influences n/ 
Mpqn the neuron output values ^re equal no matter \what ^ 
ofj^the p number of memory cells gnlgh§ have Its data y 

30 Inverted or fixed. As a result, the Influences upon 
the neuron output values due to the fall of the data 
of one memory cell can be made lower than those of the 
general binary expressions. This expression method 
will be referred to as the "equivalent expression" in 

35 the following, ^ere has been descr i be^^'the neuron v/ 



output values^ but the aforementioned equivalent ex- 
pression can naturally be used for the connection 
weights. 

The binary expressions can naturally be used. 
This case Is suited for expressing many values with 
less memory cells because the 2** values can be expres- 
sed In ^he^ p bits. Fig. 10(a) shows one embodiment of 
the case, In which the binary expressions are used for 
the neuron output values and ^e^ connect Ion weights. 
Only the memory cells on the data line of I = h (h = 
1, 2, - - -j and n) are shown In ^h'i^ array A, and only 
the memory cells on the data line of I = h and on the 
word line of s = f (f =1, 2, - - m-1) are 

shown in'^^array B. In Fig. 10(a), letters WT des- ^ 
ignate a weighing circuit for weighing the signals of 
the memory cells and transmi t*^ them to the adders ;ah 
and bh. Here, the weighing coefficient are varied for 
^h^ every memory cell^'^, as shown, from 1 to 2^ for the v/ 
neuron output values and from 1 to 2** for the connec- 
tion weights. As a result, the neuron output values 
and the connection weights to be Input^edQ to the mul- n/ 
tlpller mh can take \t^h^ 2^ and 2** numbers of values, >/ 
respectively. The address selecting method for cal- 
culating the neuron output values may follow Fig. 8(d) 
like the embodiment of Fig. 9. Fig. 10(b) shows one 
embodiment of the characteristics of the nonlinear 
transfer function circuit D in the embodiment of Fig. 
10(a). The output 'gl alternately repeats the values 0 
and 1, each time the Input varies by (x2 - xl)/2^, 
and the output g2 alternately repeats the values 0 and 
1 for a period twice as long as that of the output gl . 
The periods are likewise varied by times so that the 
output gp is set from 0 to 1 across the value of (x2 - 
xl)/2. In other words, the nonlinear transfer func- 
tion circuit D may be so designed that It may operate 



as an A/D converter. In the present embodiment, too, 
the nonl i near transfer function circuit D can be so 
designed that the neuron output values may increase 
nonlinearly for the input* In order to increase the 
neuron output values according to the sigmoid function 
in response to the Input, for example, the periods for 
the Individual outputs !lp var^ roa^ be decreased with ^ 
the increase in the input , ^hi l^'^^the period ratios be- 
tween the different values^^g being held constant, so ^ 
that the periods may be Increased with the increase in 
the Input when the Input exceeds the value of (x2 - 
xl)/2. As has been described hereinbefore, according 
to the embodiments shown In Figs* 10 (a) and 10 (b) , the 
neuron output values and the connect Ion weights can be 
expressed to have the 2^ values and the 2** values, 
respectively, by using the p and q numbers of meitfory 
cells, respectively. Thus, the embodiments are suited 
for giving the neuron output values and the connection 
weights multiple values with less memory cells. In- 
cidentally, in the present embodiment, too, various 
modifications can naturally be made such that the 
functions of the weight circuit WT and the adders al , 
- - and an, and bl, - - and bn are given to the r 
multipliers by using mul 1 1*^ I nput cl rcul ts %sy the mul-^^^ / 
tlpllers ml, - - and mn. ^ere has been described} 
the embodiments using the equivalent expressions and 
the binary notations*. In addition, there are various 
methods such as the method of expressing a negative 
number with a code bit or a method of expressing data 
in'^plural^ bl ts , which can be separately used. If nec- 
essary. 

Next, ^ere^will be described an embodiment. In ^ 
which a dynamic memory cell (I.e., DRAM cell) composed 
of one MOS transistor and one capacitor Is used in the 
memory. 



Fig. 11 shows an embodiment in which the embodi- 
ment shown in Fig. 8(a) is composed of the DRAM cells.. , 
In Fig. 11, array A and ^e\^ array B are ^ompose*§\/ 

of: a plurality of i ntersect i ng data 1 i ne pairs DAI 
5 and DAI", - - and DAn and DAn, and DBl and 5HT, - - 
and DBn and EStT, and word lines WA1, WA2,- - and 
WAm, and WBl,l, WBl . 2 , - - - WBl . n , and WB2 , I , WB2 , 2 , 
- - and WBra-l,n; and memory cells MC disposed at 
their intersections. Each memory cell MC is arranged 
10 at^i ther of) the Intersections between the paired data V 
lines and the word line. Thus, the embodiment has the 
so-called "folded-bit line structure". In Fig. 11, 
letters PR, SA, RSA and WS designate a precharge cir- 
cuit, a sense amplifier, a read sense amplifier, and a 
15 write switch, respectively, and correspond to the ar- 
ray control circuits 13A and 13B of Fig. 7(b). Lfet- 
ters MT appearing in the arithmetic circuit designate 
a multiplier. Numeral 16 designates a clock generator 
for; generating clocks *a and for controlling 

a;ther circuits in response to addresses ADDA and ADDB 
a^nd chip select signals CSA and CSB**]fiven^^ 
jijutside of the chi^. 

In the following, the operations of the embodi- 
ipent of Fig. 11 will be described with reference to 
Fig. 12 and Figs. 13(a) and 13(b). Fig. 12 shows one 
embodiment between the operation modes and the exter- 
nal, signals. As has been described hereinbefore, the 
word lines of ^^m^mory cell arrays A and B are se- 
lected one by one in the earlier half of the arith- 
metic mode, and one word line of ^he)i memory eel 1 array \J 
A is selected in the later half. In the memory mode, 
on the other hand, ^e^ memory cell arrays A and B are \/ 
independently subjected to the read^ng^ and writ(i^n^ v/^ 
operations. In Fig. 12, the operation modes are fur- 
35 ther divided so that those controls may be facilitat- 
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eds Letters AR, AW, BR and BW in the memory mode &es- 
^^^^^'^t^n^ the read^n^ mode^^rof^'\th^ array A^^lpfhe wri%n^ x/ 
mode in the array j'^he reading} mode ^ora tha^^'array B ^ 
i.nd^^e writ^^^mode in the array B, respectively. On 
5 the other hand, letters NR and NW i n the arithmetic 

mo<fe^^the earlier half for reading and calculating the 
individual data and the later half for writing the 
arithmetic resul ts, respect I vely. In order to switch 
those six modes, the present embodiment uses four ex- 

10 ternal input signals: chip select signals CSA and CSB; 
write control signal TTE; and arithmetic circuit con- 
trol signal TJe'. The chip select signals CSA and CSB 
assign the selections of the chips and the arrays A 
and B. The chips are unselected if both the signals 

15 CSA and CSB are at the H Cl-e., high) level; the array 
Is selected If the signal CSA Is at the L (i^e., /low) 
level; and the array B Is selected If the signal CSB 
Is at the L (l^e., low) level. The write control sig- 
nal Wis ^ej^for switching ^he\^wr I te and read, as has 

20 been described hereinbefore, and establishes the read- 
ing operation at the H level and the writing operation 
at the L level. The signal fJ^ also establishes the 
memory mode at the H level and the arithmetic mode at 
the L level, as has been described hereinbefore. If, 

25 Shereforl* both the signals CSA and CSB are at the L ^ 
level and the signal WE Is at the H level whereas the 
"signal NE is at the L level, gor example^ |here is es- 
tabli.shed^ the earl Her half mode NR of the arithmetic 
raode^ In which bo th it_h^ arrays A and B are read out, 

30 Since |.h^ switching of ^le^ arrays A and B is assigned ^ 
by the chip select signal, the address signal can 
divide the addresses Into the address group ADDA for 
selecting the memory cells of the array A and the 
address group ADDB for selecting the memory cells of 

35 the array B. Here, the address group ADDA is the 
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generic name of the X-addresses for selecting the word 
lines of the array A and the Y-addresses for selecting 
the data lines of the array A, Likewise, the address 
group ADDB Is the generic name of the X-addresses for 
5 selecting the word lines of the array B and the Y- 

addresses for selecting the data lines of the array B. 
In each operation mode, these address groups are ap- 
plied to the address pins in accordance with Fig. 12. 
According to the embodiment of Fig* 12 thus far de- 

10 scribed, the two chip select signals are provided to 

switch |h^ arrays A and B thereby^Q^ separatfe the ad- ^ 
dresses between ^h% arrays A and B, Since arrays \/ 

A and B "can be Independently selected , 1 1 I s possible 

to control each operation mode easl ly ^o^* selecting ^ 

15 ^eJarrayf^A and/^B* Incidentally, the rel at I onD be- 
tween the operation mode^l and the external signal'^ can 
naturally be modified In various manners In addition 
to those of Fig. 12. For example, \here can b^'^adoptt ^ 
ed^: the method of adding addresses for switching the ^ 

20 arrays A and B by using the chip select signal CS 

only; or the method of generating the X-address for 
selecting the word line of ^h^ array B In the mode NR ^ 
from the counter disposed In the chip, by not dividing 
the addresses for ^h^ arrays A and B but selecting ^ 

25 either array A or B only. 

Fig. 13(a) shows an embodiment of the waveforms 
of Fig. 11 In the memory mode, and Fig. 13(b) shows an 
embodiment of the Waveforms of Fig. 11 In the arith- 
metic mode. 

30 The operations of the memory mode are similar to 

the read^n^ and wr I t^g\ operat I ons of the ordinary 
DRAM. Fig. 13(a) shows the voltage waveforms In'^case 
the read^n^ operations ( In the mode AR) and the wrilt ^ 
ingj operations (In the mode AW) are continuously ex- ^ 

35 ecuted for the memory cell at the Intersection between 
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the word line WAl and the data line DAI In ^hj array A 
m the memory mode. In Fig. 13(a), letters Vcc desig- 
nate a positive supply potential. Since, in the 
memory mode, the ar I thraet I c ci rcui t control signal TJE 
5 Is at the high level, the arithmetic circuit start 
signal 4> N is fixed at the low level so that the a- 
rlthmetlc circuit 12 Is OFF. Before the start of the 
reading operation, signals PPA and PNA are set (lat]^^ x/^ 
Vcc/2 so that the sense amplifier SA is OFF. Since a 

10 precharge signal * i»/w Is at the high potential, on the 
other hand, the precharge circuit PR is turned on to 
short the data line pairs DAI and DAI , - - and DAn 
and 5An and to set the potential at a precharge poten- 
tial VH- This precharge potential VH is set^^t^ Vcc/2 

15 according to the so-called "half -precharge method". 

When the chip select signal CSA drops to the low ^po- 
tential, the precharge signal * pa falls to turn off 
the precharge circuit PR so that the word line WAl 
selected by the address signal ADDA and a read Y- 

20 Select signal YRAl are transited to the high poten- 
tial. As a result, the MOS transistors of all the 
memory cells MC connected with the word line WAl are 
rendered conductive to^establish \lndl vidual 1>^ del Icate 
potential differences on the data line pairs DAI and 

25 DAI , - - and DAn and DAn in accordance with the 

electric charges stored In the capacitor. This poten- 
tial difference is read out and is detected by the 
sense amplifier RSA' f ed wl th the Y-Select signal YRAl 
so that It Is converted Into the impedance difference 

30 of read lines OA and OA. This Impedance difference is 
converted by the Input/output circuit Into a voltage 
difference, which Is amplified so that the content of 
the memory cell. I.e., the potential corresponding to 
1 or 0 is output^e^ as the read data DO, The so- 

35 called "rewriting operation" is executed in parallel 
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with the aforementioned operations In the following 
manner. After the Individual delicate potential dif- 
ference^'^ have been established in the data line pairs 
DAI and DAT. - - and DAn and DAn, the siisnal PPA is 
transited to the high potential whereas the signal PNA 
is transited to the low potential to start the sense 
amplifier SA. As a result, the delicate potential 
difference established In the data line pairs Is am- 
plified to transit the data lines at the high poten- 
tial to the level Vcc arid the data lines at the low 
potential to 0 V. As a result, the capacitors of all 
the memory cells MC connected with the word line WAl 
We wri t t en~^^ wi th the po ten t i al correspondi ng to 

the data before read. When the chip select signal CSA 
takes the high potential after the end of the rewrit- 
ing operation, the selected word line WAl and the read 
Y-select signal YRAl are transited to the low poten- 
tial, and the signals PPA and PNA are then transited 
to Vcc/2 to turn off the sense amplifier SA and tran- 
sit the precharge signal 4> ra to the high potential. 
As a result, the data line pairs are shorted, and the 
precharge potential VH Is set with the potential, un- 
til the Initial state Is restored. The operations 
thus far described are the reading operations. 

Subsequently, the operations are shifted to the 
writing operations (In the mode AW) of the same cells. 
When, In the writing operations, the chip select sig- 
nal CSA takes the I'ow potential and the write control 
input WE takes the low potential, the data given to 
the write data DI are written In the selected memory 
cell In the array A. In the writing operation, too, 
the precharge signal ^pa is dropped at first to turn 
off the precharge circuit PR when the chip select sig- 
nal "CSA drops to the low potential. Next, the word 
line WAl selected by the address signal ADDA and the 



Y-Select signal YRAl are transited to the high poten- 
tial. As a result, the MOS transistors of all the 
memory cells MC connected with the word line WAl are 
rendered conductive so that the delicate potential 
differences are individual 1:^ es tabl i shed*^ i n the data 
line pairs DAI and DaI , - - and DAn and DAn in ac- 
cordance with the electric charges stored in the capa- 
citor. The delicate potential differences established 
in the data line pairs are amplified by the sense 
amplifier SA. Subsequently, an input circuit starting 
signal * WRA generated as a result of ^ trans i t i on of 
the control input WE to the low potential is transited 
~~ t o th e h t gh po fen t l a 1 . : As a res ul t r th e -da t a g i ven to 
the write data DI are transmi t ted as the difference^ 
signals to write line pairs lA and TX- Moreover, the 
write Y-select signal YWAl is transited to the hVsh 
potential to turn on the write switch WS connected 
with the write destination memory cell. As a result, 
the write line pairs lA and lA are conducted to the 
data line pairs DAI and DAI , respect ively. < As a re- 
sult, the data line pairs DAI and DAI are set to the 
potential corresponding to the data fed to the write 
data DI . After this, the Input circuit starting sig- 
nal * Is transited to the low potential, but the 
potential of the data line pairs Is held by the sense 
amplifier SA. In the data line pairs for which the 
write switch WS Is not turned on, the signal read out 
at first Is aropllfl'ed as It Is by the sense amplifier 
so that the rewrite Is executed. When the chip select 
signal CSA takes the high potential after the end of 
the rewriting operation, the selected word line WAl 
and the write Y-selectlon signal YWAl are shifted to 
the low potential. After this, the signals PPA and 
PNA are transited to Vcc/2 to turn off the sense 
amplifier SA and to transit the precharge signal * i-a 
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to the high potential. As a result, the data line 
pairs are shorted and \l}as It^-'potentlal set to the ^ 
precharge potential VH so that the Initial state is 
restored. The operations (Uius fag descr I bed*^ are the v/ 
5 writing operations. 

Here, the foregoing description Is directed to 
the case in which a memory cell in the array A is con- 
tinuously subjected to the reading operation and the 
writing operation. Despl te ^f^this description, how- J 

10 ever, one of the reading and writing operations can be 
continuously executed. Moreover, It Is quite natural 
that memory cells In desired positions in a memory 
eel 1 "a3^r^y» which are different for every reading ope- 
ration or writing operation, can be subjected to the 

15 reading operation or the writing operation by switch- 
ing the modes AR, AW, BR and BW. 

Next, the operations of the arithmetic mode will , 
be described Un the following. Fig. 13(b) shows the ^ 
operation waveforms for obtaining a neuron output . 

20 value Vxa. (Let 1 1 bej jiissume^) that the necessary con- J 
nectlon weights and neuron output values or the like 
have already been written by the writing operations in 
the memory mode. In order to establish the mode NR, . 
first <^ al?^, the chip select signals CSA and CSB are J 

25 set to the low level, and the write control signal WE 
Is set to the high level whereas the arithmetic cir- 
cuit control signal NE is set to the low level. The 
addresses ADDA and 'ADDS are so set as to select the 
word line WAl of the array A and the word line WBl of 

30 the array B. Since the signals CSA and CSB are at the 
low level, the precharge signals * j.^ and 4> i>» are 
transited to the low level. Since the signal NE is at 
the low level, the arithmetic circuit starting signal 
* N is transited to the high level. Subsequently, the 

35 word lines WAl and WBl are selected so that the neuron 



output values Vxi, Vai, - - and V„i and the connec- 
tion weights T^xi, T%a. . and T^i„ are read out 

of the memory cells fei^ the word line WAl onto the data 
lines. Thus, the neuron output values read out from 
the array A and the connection weights read out from 
the array B are input^ed^ to the multiplier MT which \/ 
has been started by the arithmetic circuit starting 
signal 4> N, as shown in Fig. 11. In the multiplier 
MT, the data lines at the side of (Ih^ array A and the 
data lines at the side of [Ih^ array B are Individually 
connected with the gates of the MOS transistors (Ti, 
Ta), which In turn are connected with the product/sum 
output line NO— and the dummy line DM -through the swi- 
tching MOS transistor (Ta) fed with the arithmetic 
circuit starting signal * n. The product/sum output 
line NO has Its one terminal connected with the ]»ower 
source VM through the load RMl , and the dummy line DM , 
has Its one termi'Kar ear th^d to the groun<|. When the ^ 
signal read out to the data lines Is amplified to Vcc 
or 0 V by the sense amplifier SA, the electric current r 
\Xow^"£hrough the load RMlJ from the power soutce VM, to v/ 
the earthed^ electrode In the mul tipl ler^ In which the ^ 
product |,6\^the neuron output value and the connection 
weight is 1. As a result, the potential of the pro- 
duct/sum output line NO drops In proportion to the 
number of combinations. In which the product of the 
neuron output value and the connection weight Is I. 
The product/sum NO 'Is ihput^ecQ to the nonlinear trans- \/ 
fer function circuit D. In this nonlinear transfer 
function circuit D, the summation of the products of 
the neuron output values and the connection weights is 
so high that the detection result of whether or not 
the potential of the product/sum output line NO i s ^ 
lower than reference voltage VR Is output^ed^ to ^3 line y 
NV. In the waveforms of the product/sum output line 
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NO shown in Fig. 13(b), solid lines show the case^i^in ^ 
which the result of the product sum is smal IjV^ fe.nd\ v/ 

Jvv^^^^A^broken curves show the case in which the result of the ^ 

product sum is large. The input/output circuit de- 
5 tects the result of the nonlinear transfer function 
circuit D and outputs the neuron output value Via, 
which is to be subsequently written in the memory 
cells, to the write line pairs lA and TA- Fig. 13(b) . 
shov^ the waveforms of the write line pair lA. This \/ 

10 pair lA takes the high level, as indicated by a broken 

curve. in'^aseAthe product sum is large, and the'' low ^ / 
Qvel\ as indicated by a solid curve^^.^n'^^ase'Hhe pro- \J 
duct sum is small, j?he word line pair lA takes an op- V 
posite phase. At the time when the neuron output / 

15 values are output^e^ to the write line pairs lA and V , 
Qi^.^^a latch signal $ i- is transited to <^h^ high poten- 7 
tial. As a result, the potentials output^e^^ to the ^ 
write line pairs lA and Ta are latched by a latch cir- 
cuit which is disposed in the input/output circuit 10. 

20 The latch signal * u may be^ised with^ del£3^\UllJ 

the signals appear in the pairs lA and lA, in response 
to the fall of the arithmetib circuit starting signal 
NE. Subsequently, the arithmetic circuit starting 
signal * N is transited to the low potential to turn 

25 off the arithmetic circuit so that the data lines are 
precharged as in the memory mode after the word lines 
have fallen. At this time, the latch signal * ^ is 
left at the high potential so that the neuron output 
values output^e^ to the write line pairs lA and lA are ^ 

30 held^ A/V / 

Next, the mode shifts to the* mode ^^^t i.e., the 
later half of the arithmetic mode. First ^f allj, the ^ 
chip select signal CSA and the write control signal WE 
are set to the low level, and the chip select signal 

35 csb" is set to the high levef^to switch the address 



ADDA so that the memory cell for writing the neuron 
output value in the array A may be selected. The 
arithmetic circuit starting signal NE is left at the 
low level. As a result of the fall of the signal CSA, 
the precharge signal 4» drops to the low level to 
establish a state'^^or thel array A to be written. Sub- 
sequently, the potentials of the selected word line 
WA2 and the write Y-selectlon signal YWAl fall. As a 
result, the neuron output value Viaj outputjte^ to the 
write line pairs lA and Ta^ i s wr 1 1 ten In the memory 
cell which is connected wi th ^hej WA2 and the data line 
DAI. Finally, all the potentials of the word lines 
are dropped for~~the "precharge : Si nee , moreover , the 
arithmetic circuit control signal NE falls, the latch 
signal * I. falls to release the latch. Thus, prepara- 
tions are made for the next operations. The opepa- 
tions ^us fa^ described'* are'' thos^ in the arithmetic 
mode. All the neuron output values can be calculated 
by continuing similar operations in different addres- 
ses in accordance with Fig. 8(d). 

In the structure thus far described, the circuit 
of the multiplier MT, which is connected with the 
dummy line DM, may be omitted. If, however, the gate 
capaci tors^ or the 1 ike,^fj'* the MOS transistors of the 
multiplier MT are added to the data lines only at ^one 
side, the data line capacities are unbal anced (tcjArou- 
bl^ the operations of the sense amp 1 1 f I er fesj the case 
may be. ^n this cafee, the structure of Fig. 11 could 
avoid) the inferior Influences coming from the unbal- 
ance of the data line capacities'. 

Next, an embodiment of the circuit suitable for 
use In Fig. 11 Is shown. Fig. 14(a) shows one embodi- 
ment of the nonlinear transfer function circuit D. 
The present embodiment Is constructed of: a differen- 
tial amplifier composed of bipolar transistors Q720 



- 46 - 



and Q719, a resistor R72 , and a MOS transistor Q721; 
and an inverter composed of an inverter INV75, MOS 
transistors Q715t Q716, Q717 and Q718, a resistor R71 
and a diode D71, The present circuit is** started when 
the signal *n takes the high potential. Fig, 14(b) 
shows a relation between the potential of the product/ 
sum output line NO^(o^ the input of the nonlinear 
transfer function circuit Dy and the potential of the 
output NV, The output NV takes the high potential, if 
the potential of the product/sum output line NO is 
lower than the reference potential VR, but takes the 
low potential if the potential of the line NO is 
—higher- than the reference po ten t i al VR. - Since, ac- 
cording to the present embodiment, ^e\ bipolar tran- 
sistors are used in the differential amplifier, it is 
possible to realize a nonlinear circuit which is/ 
characterized to have a steep rise for the change in 
the input. By setting the reference potential VR to a 
desired value, moreover, the characteristics of the 
nonlinear transfer function circuit D can be easily^ 
changed. Incidentally, the output of the different'*^ 
amplifier cannot be made so high so as to avoid the 
saturation of the bipolar transistor Q719. (As^a re- 
sult, the downstream inverter may not operate if it is 
connected directly with the output of the differential 

amplifier. Therefore, the resistor R71 and the diode 

*^^t ^/icirh** r 

D71 are provided to groigfthe po ten t i al^" to be Inputytedj 

to the MOS transistor Q717^l^^^^' 

Fig, 14(c) shows one embodiment of the input/out- 
put circuit 10. The write circuit WR Is composed, as 
shown In Fig. 14(c)., of an Input buffer INBUF, write 
switches SWA and SWB, a latch circuit LAT , and In- 
verters INVIA and INVIB. The write switches SWA and 
SWB are used to swi tch'^^hich of the\ arrayfej^ A ^nd^ Bj has 
its memory cells written with the write data DI • When 
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the switching signal * wra is at the high potential, 
the write data DI Is written through the input buffer 
INBUF ^i^Hhe memory cells of ^hei array A by the write ^ 
line pairs lA and lA. When the switching signal 4> wr3 
5 is at the high potential, the writj data DI is written 
through the input buffer INBUF |_n^''the memory cells of ^ 

array B by the write line pairs IB and IB. The \/ 
latch circuit LAT latches the data, which are out- 
putted) to the output NV of the nonlinear transfer ^ 

10 function circuit D In the arithmetic mode, to write 

them in the memory cells of array A by the write n/ 

line pairs lA and LA. Since the potential rel at I ons^^ v/ 
of the output NV" of the -nonl i-near -transfer funct i on 
circuit D and the write line pairs lA and lA&r^Mn 

15 phase, ^sj^'is apparent fromn4(c) , the potential rela- v/ 
tion^'^between the product/sum output line NO of ^tie v/^ 
nonlinear transfer function circuit D and the input 
common line I A ^rei^'oppos I te'' (to eacB othe?i, as shown In v 
Fig. 14(d). Since, in the embodiment of Fig. 11, the 

20 potential of the product/sum output line NO Is the 

p 

lower ^o^^the larger product sum of the neuron output 
values and the connection weights, as has been de- 
scribed hereinbefore, the circuit is made such that 
the potential relation^ between the product/sum output 
25 line NO and the Input common line I A (arej^oppos i te to ^ 
-'^^n'^^ach otheR.5^"'^n^case the potential of the product/sum 

yo increase / 

output line NO I s \so\ des Igned'^ that It may*Qrlse thel ^/ 
more for larger' product sunf* of the neuron output v/ 

values and the connection weights, it Is quite natural 

30 that the cl rcul t ^a>3 be (so] made'^^'that the potential 

rel at ion^'** between the product/sum output line NO and\/f 
the input common line lA^rS^'^ln phase. 

Fig. 14(e) shows one embodiment of the read cir- 
cuit OUT, This read circuit OUT is composed of a 

35 current/voltage converter IVOUTl, a level shift cir- 
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cuit LS, a read latch circuit OUTLT and an output buf- 
fer BUFOUT, In the current/voltage converter IVOUTl, 
the data read out as the impedance differences to the 
read lines OA and OA and the read lines OB and OB are 
5 converted to the differences of the voltages of the 

lines OA* and OA* ^fj^ead lines OB' and OB* . In the J 
level shift LS, the voltage of the data read out from 
the current/voltage converter IVOUTl is shifted to a 
level, at which the bipolar transistors in the down- 

10 stream read latch circuit OUTLT are not saturated, to 
transmit it to the read latch circuit OUTLT. 

A detailed embodiment of the read latch circuit 

OUTLT is shown in' Fig. 14(f). The read differential 
amplifiers AMPA and AMPB in the read latch circuit 

15 .OUTLT are used to swi tch ^hi ch of3 the data read out ^ 
from the memory cells of^h^ array A through the read 
lines OA and OA to lines LI and L2 \an^^the data read ^ 
out from the memory cells of ^h^ array B through the \/ ^ 
read lines OB and OB to the^ lines L3 and L4f'^re to be ^ 

20 read out as the read data DO. When a switch signal * 
A is at the high potential, the data read out from the 
memory cells of ^h^^ array A are output^ed^ as the read ^ 
data DO. When a switch signal * b Is at the high 
level, the data read out from the memory cells of ^e^ ^ 

25 array B are output^d\ as the read data DO. In the ^ 
read latch circuit OUTLT, the bipolar transistor QIA 
is turned off, but the bipolar transistor QIB is 
turned on when the Vead latch circuit * is tran- 
sited to a higher potential than the voltage Voa* As 

30 a result, the differential amplifiers AMPA and AMPB 

are turned off, and the differential amplifier AMPC is 
turned on. As a result, the read data are latched by 
the differential amplifier AMPC and the level shift 
circuit LSC. Specifically, according to the present 

35 embodiment, the read data DO can be latched and ^on- 
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t inuouslPi output^e<^ for a des i red'' t Ime (per i od] by tran- ^/ 
siting the read latch circuit * to a higher poten- 
tial than the voltage Vsa after the read data have 
been fixed. 

5 Incidentally, in the embodiment of the multiplier 

MT shown in Fig. 11, the data line pairs of array ^ 

A are connected with the gates of the MOS transistors 
which are farther from the earthed electrode than the 
data line pairs of array B. As a result.'^the neu- v/ 

10 ron output values and the connection weights are not 
^i^'^equivalentlyCbandled when a product is to be taken], i/ 
If this raises a problem, &i|^mbodiment ^g^Fig. 15 may ^ 
be used. In Fig. 15, the da ta- 1 i ne DAi i s connected 
with the gates of MOS transistors Q7C3 and Q7C6 , and 

15 th^A^data 1 i ne (pai ?^ DBi is connected with the gates of ^ 
MOS transistors Q7C5 and Q7C4. Since the two dat/a 
lines are connected with the MOS transistors closer to 
the earthed electrode and the MOS transistors farther 
from the same, the neuron output values and the con- / 

20 nection weights ar^e'^^ui val en 1 1 y handled. As has been \/ 
described hereinbefore, according to the embodiment 
shown in Fig. 11, the embodiment shown in Fig. 8(a) 
can be realized by using the DRAM cell which is com- 
posed of one MOS transistor and one capacitor. the 

25 DRAM cell can have its occupied area dras t i cally 

reduced^to a t tai n (a mer i'l^Mhat i t \ca^ be ^eal i zed in^ \/ 
higji* integra^tonl over the chip. Mn the DRAM cell com- 
posed of one transits tor and one capacitor, (although \/ 
not touched in the foregoing description^) refreshing v/ 

30 operations are necessary within a constant time period 
for compensating the reduction of the stored charges 
due to the current leakage of the capacitor. In the 
present invention, too, the refreshing operations can 
be easily accomplished, If necessary, like the ordi- 

35 nary DRAM no matter which ^h^ mode might b^^ ^he memory ^ 



9 ' - 50 - 



mode or the arl thme 1 1 c mode • ^en^ 

• In the Aforementioned)'^ embodiment , the memory 
cells are exemplified by the DRAM cells but should not 
be limited thereto,^ bu£^ s iml 1 ar data processing sys- v/ 
tem can also be realized by using even other memory 
cells. Next, an embodiment using SRAM cells will be 
described \j_n the followln|^ Figs- 16(a) and 16(b) are 
circuit diagrams showing SRAM cells MCS. In the em- 
bodiment of the present Invention, the SRAM cells MCS 
are used as the MC of Figs, 7(a) and 7(b). Fig. 8(a), 
Fig. >(a) and Fig. 10(a). Either of the embodlmentsy . 
Co^'^Flgs. 16(a) and 16(b), is advantageous in that the ^/ 
controls are easier than the case^'Iof using the] \/ 

DRAM cells because it requires nei ther'^ rewri 1 1 ng nor 
refreshing operat I on \unl Ike the DRAM cells]. Fig. v/ 
16(c) shows one embodiment for realizing the embodi- 
ment of Fig. 8(a) by using the SRAM cell of Fig. 16(a) 
or 16(b). In Fig. 16(c), letters MCS designate the 
SRAM cell, and letters LD designate a data line load. 
Figs. 17(a) and 17(b) show examples of the operation 
waveforms. Fig. 17(a) shows an example of the caseQ v/ 
in which cells connected with the data lines DAI and 
DaT and the word line WAl are continuously subjected 
to the reading (operatlonsi and ^hej wrl ting operations ^ 
in the memory mode. Fig. 17(b) shows an example of 
the operation waveforms of the case^^ In which the v/ 
neuron output valve Vxa Is to be calculated in the 
arithmetic mode fro'm both the neuron output values 
Vii, Vai, - - -. and V„Vstored in the memory cells of 
the word line WAl^ and the connection weights T^xx, ^ 
T^ax, - - -» and x^iJi'^Vtored in the memory cells of ^ 
the word line WBl . The basic operations are similar 
to those of the aforementioned case of the DRAM cells, 
and their description will be omitted. The SRAM cell 
has ^ meri^^^hat Its control Is simpler than the DRAM 
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cell, because It does hot need the rewriting operation y 
and the refreshing operation. Because (jg^^o necessity ^ 
for the rewriting operation, moreover, ^here IsJ 
another (meri^^^tl&at the reading and writing speeds in / 
5 the memory mode and the cycles in the arithmetic mode 
can be accelerated. 

The description thus far made is directed to the 
example of the circul t s tructure, in which the embodi- 
ment of Fig, 8 Isv real ized by using the DRAM cells and 

10 the SRAM cells. Next , ^ere wl 1 1 be described} an ex- \/ 
ample of the circuit structure for expressing the neu- 
ron output v^l^^s^*^^ ^^^^^g^^J^^ weights by using 
a plurality of memory eel IsV Although the embodiment 
to be described uses^h% DRAM cells, the present in- ^ 

15 vention can 1 ikewi se'^^real I zed even lS>rJ usi ns Qe^ \X 
SRAM cells. / 

Next, (here will be describedjan example of the 
circuit structure In which the neuron output values 
are expressed by using the DRAM cells and In which the 

20 connection weights, f re expressed by using a plurality 

of memory cellsl In Fig. 18(a), the data line pai rs ^ 
DAll and DAlT, DA12 and DA12, - - r, and DAIP and DAIP 
In ^h"^ array A correspond to the data line pairs In ^ 
^h^ array A, which are to be Input^^e^ to the adder al v/ 

25 in Fig. 9(a). On the other handi the data line pairs 
DAnl and DAnl , and DAn2 and DAn2, - - and DAnP and 
DAnP In \|hei array A correspond to the data line pairs >/ 
In ^he\ array A, whl'ch are to be Input^ted^ to the adder y 
an in Fig. 9(a). (jh^ ^rray B has similar correspon- V 

30 dences. As shown in the input/output circuit DIOlO, 

\there are provided an] r number of Input terminals DOl , \X 
- - and DOr and ^nl r number of output terminals i/ 
Dii, - - -, and Dlr^*? wherel n r Is the larger number of 
P and q) so that <^hej da£li"'^^of p bits or q bits Indlcat- 

35 ing the neuron output values or the connection weights 
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may be simultaneously read out or written in^h^ memo- >/ 
ry mode. In array A in the ar I thmet ic mode, the y 

data of every p bits read out to the data lines by 
selecting the word lines are synthesized by the adder 
ADD to output the neuron output values to the neuron 
output value output lines VOl, V02 , - - -,^and^YOn. 
In ^hej array B, on the other hand, "VLh^ daTa'^ of q bits ^/ 
read out to the data 1 Ines by select Ing the word lines 
are synthesized by the adder ADD to output the connec- 
tion weights to connection weight output: 1 Ines TOl, 
T02, - - and TOn. These values are input(te^ to the ^ 
BLK2 so that the resultant product sum Is inputljgd^to ^ 
the nonlinear transfer function circuit DIO- The out- 
put of the nonlinear transfer function circuit DIO 
corresponding to the neuron output value is trans- 
mitted to the input/output circuit DIOlO and latched 
by the latch signal * Subsequent ly, the address is 

switche^^to select the p number of cells to write the ^ 
determined neuron output value, and the write Y-select 
signal YWAi is raised to write the latched neuron out- 
put values in parallel in the p number of selected 
cells. By continuing these operations, the neuron 
output values can be updated like*^the embodiment of v/ 
Fig. 11(a). According to the present embodiment, the 
embodiment of Fig. 9(a) can be rea^lized by equivalent- 
ly adding the data of the plural^^memory cells input^ed)^ 
to the adder ADD. By weighing and adding^bit by bit, 
the data of the pi u'rai^ memory cells Input^e^ to the >/ 
adder ADD, moreover, it is possible to realize the em- 
bodiment of Fig. 10(a), in which the neuron output 
values and the connection weights are expressed with 
binary numbers of^pluraT bits. Since the present em- ^/ 
bodiment can also be applied to the case in which the 
neuron output values and the connection weights are 
expressed with a plurality of bits by another method. 



a variety of data processings can be accomplished in 
accordance wl th the purpose. Since the DRAM cells are 
used in the present embodiment, a high Integration can r 
be achieved. Since, moreover, the data; of the plural^' 
memory cells are processed in paral lely bo th in the ^ 
memory mode and In the arithmetic mode, ^h^ data pro->/ 
cesslng can be executed at ^"^high speedf^ 1 ike'-? the case >/ 
^g'^express ions Qfl tlf^ 1 bit, although the neuron output </ 
values a^nd the connection weights are expressed by the^ 
plurai^bits. Here, in ^he"}^ BLKl , the signals of^*plural^ >/ 
memory cells are synthesized by^the adder, and the 
resul t is input^^ to \th^ BLKZ'^actll^g^ as the product/ 7 
sum circuit. However, a variety of modifications can 
be made by omitting the ^addition at BLKl ^ucAjl^ that >/ 

the data of the pi ural^memory cells indicating the ^ 
neuron output values or the connection weights ai^fe in- 
put^ted^ in parallel to the product/sum circuit BLK2 so 
that they may be subjected to \l^h^ multiplications and v/ 

summations. boJ»'^*' / 

In the following^. Fig. 18(b) for realizing the ^ 
embodiment of Fig. 9, in which the neuron output 
values and the connection weights are expressed in a 
plurality of equivalent bits by the embodiment shown 
in Fig. 18(a), shows one embodiment of ^h^ BLKl of ^ 
Fig. 18(a). ^re is shown^lhel BLKi^^t^hTch is connected 
with (th3^ data lines DAll, - - -, and DAIP of {th§ array v/ 
A. The same circuit can also be used in another BLKl 
of (th^ array A. Thfe circuit of the present embodiment \/ 
can also be used in [th% array B, if the number of the n/ 
data line pairs, the read line pairs or the write line 
pairs is changed from p to q and if a q number of cir- 
cuits, each having a p number of precharge circuits PR, v/ 
are provided. In the present embodiment, (there are 
provided^ p pairs of read line pairs OAl and OAl , - - 
and OAp and OAp and p pairs of write line pairs lAl 
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and Tai, - - and lAp and lAp so that the p number ^ 
of memory cells may be subjected to the writing or 
reading operations in parallel* The read sense ampli- 
fier RSA and the write switch WS are consecutively 
connected In the same BLKl , as shown, with the read 
line pairs OAl and OAl i - - and OAp and OAp and the 
p pairs of write line pairs lAl and lAl, - - -» and 
lAp and lAp. For one pair of read or write lines, 
specifically, ^ver:^^p pai r^ ar^*^ connected with the ^ 
data line pair. The adder ADD is composed of a load 
circuit LD103 and a p number of vo 1 tage/curren t con- 
verters VI. In the voltage/current converter VI, the 
"data lines D^^^^ -» and DAlp are connected 

with the gates of the MOS transistors, which in turn 
are connected In series with the MOS transistors hav- 
ing their gates fed with the ar i thme t ic circuit start- 
ing signal * N, to connect the earthed electrode and 
the neuron output value output line VOL This neuron 
output value output line VOl Is connected through a 
resistor in the. load circuit with the power source 
VMOl. As a result, if the amplification of the data 
line potential %s ende^'^in the state started by the 
arithmetic circuit starting signal * n, the potential 
of the neuron output value output line VOl Is dropped 
by a voltage proportional to the number of data lines 
which are raised to the high potential, i.e., Vcc. 
According to the present embodiment, therefore, the 
neuron output value's can be expressed in terms of the 
potential drop of the neuron output value output line 
VOl. Incidentally, the provision of similar cl rcul ts 
at one side of the data lines DAll . - - -, and DAlp is 
to avoid the unbalance of the data line capacities for 
the same reasoning as that of the multiplier MT of 
Fig. 11(a). According to the embodiment thus far de- 
scribed, the neuron output values or the connection 
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weights expressed by the plurar memory cells can be ^ 
read out to the neuron output value output line or the 
connection weight output line. 

Flff. 18(c) shows one embodiment of ^% block BLK2 ^ 
5 for calculating the product sum of the neuron output 
values and the connection weights and the nonlinear 
transfer function circuit DIO. In Fig. 18(c), ^e\ v/ 
block BLK2 is composed of the load circuit LD102 and 
the multiplier MTIO, The neuron output value output 

10 lines vol, V02 , - - and VOn and the connection 

weight output lines TOl , T02 , - - and TOn are con- 
nected with the gates of the MOS transistors MiScl and 
~ M16c2 of the HTrO", and the MOS transistors are con- 
nected in parallel wi th the MOS transistor M16c3, 

15 which has its gate fed with the arithmetic circuit 

starting signal 4> Nt to connect the earthed electrode 
and the product/sum output line NO, On the other 
hand, the product/sum output line NO is connected 
through the resistor R02 In the load circuit LD102 

20 with the power source VM02. In the state In which the 

arithmetic circuit starting signal * n Is ^t th^ high k/ 
{eve^,so that the present circuit Is started, the ^ 
potential of the product/sum output line NO is dropped 
(the more^ for the larger sum of the products of the 

25 potentials of the corresponding neuron output value 

output lines VOl , V02 , - - and VOn and the connec- 
tion weight output lines TOl, T02 , - - and TOn. As 
has been described hereinbefore, the potentials of the 
neuron output value output lines VOl, V02 , - - -, and 

30 VOn and the connection weight output lines TOl, T02 

and TOn are dropped substantially In proportion to the 
magnitudes of the neuron output values and the connec- 
tion weights so that the potential of the product/sum 
output line NO becomes \th^ higher for the larger pro- \/ 

35 duct sum of the neuron output values and the connec- 
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tion weights. The produc t/sum output line NO is in- 
put(te^ to the nonl i near transfer function circuit DIO, 
The nonlinear transfer function circuit DIO can be 
constructed by connect ing ^fJ^ n number of circuits ^/ 
5 shown in Fig. 18(d) in parallel. The circuit of Fig. 
18(d.) is made like the nonlinear transfer function 
circuit D of Fig. 14(a) by combining the differential 
amplifier and the inverter; Since, however, the 
polarities of the product/sum output line NO and the 

10 product sum of the neuron output values and the con- 
nection weights are different between the embodiments 
of Fig. 11 and Figs. 18(a). 18(b) and 18(c), the re- 
sistor Rx of the differential amplifier of Fig. 18(d) 
is connected in the''''pos i t i on ^posedj to ^hat of th^ v/ 

15 resistor R72 of Fig. 14(a). In Fig. 18(d), therefore, . 
the output NVx lransit%^"to the high potential if dhe ^ 
product/sum output line NO exceeds the reference volt- 
age VRx (X = 1. 2, and p). I f [a^ P number of / 

such nonlinear transfer function circuits DSx are 

20 provided and if the reference voltage VRx is changed, 
as shown in Fig. 18(e), the change in the product/sum 
output line NO can be indicated by the number (of such \/ 
one^ of the p outputs NVx i.^'^take the high potential, v 
According to the present invention, the characteris- 

25 tics of the nonlinear transfer function circuit can be 
easily varied by varying the value o:^ the reference 
voltage VRx. I nciden tal ly , i n^'^case'^ the circuit shown ^ 
in Fig. 18(c) Is u^ed as the multiplier MTIO, the 
potential variation of the product/sum output line NO 

30 Is generally kept away from linearity for the mag- 
nitudes of the product sum of the neuron output values 
and the connect ion weigh ts by the characteristics of 
the MOS transistors. It Is, therefore, advisable to 
set the value of the reference voltage VRx by con- 

35 slderlng the characteristics of the multiplier or the 



adder so that the characteristics of the nonlinear 
transfer function circuit may take a desired shape. 
As the case may be, the characteristics of the in- 
dividual chips may be made difficult to understand ac- 
5 curately because of the fluctuations of the production 
conditions. In this case, the known neuron output 
values and connection weights are actually written In 
^e) arrays A and B, and the potential of the product/ x/ 
sum output line NO in the arithmetic mode Is measured 

10 so that the vale of the reference vo 1 tage VRx may be 
resultantly trimmed to the desired characteristics. 

Incidental ly, Qiere will be omitted] the detail^ of ^ 
the input/output- circuit DIOlOof Fig, -18(a)\ The ^ 
circui t fefl^reading or writing a plurality of memory ^ 

15 cells in parallel can be easily realized by using a 
plurality of circuits which are similar to the rfsad 
circuit OUT or the write circuit WR shown In Figs. 
14(c), 14(e) and 14(f). Moreover, the structure of 
the clock generator 16 will be omitted but can be 

20 easily realized like the circuit used In the ordinary 
memory. 

Next, the method for realizing the embodiment of 
Fig. 10, m which the neuron output values and the / 
connection weights are binarlly expressed In^plural* 
25 bits, will be described In connection with the embodi- 
ment of Fig. 18(a). In or^der to add the data expres- 
sed binarlly wl th'^plural^^bi ts , as shown in Fig. 10(a), J 
it Is necessary to 'weigh^^ and add the data of the \/ 
plurar-^memory cells bit by bit. For th i s^'fhecess i ty), 
30 ^.the potential of the neuron output value output line ^ 
vol drops in proportion to the magnitude of the binary 
neuron output valueSflif the ratios oT the ^gate^ wld uT^v/ 
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of the MOS transistors connected with the data lines 
in the voltage/current converters VII, VI2, - - -, andj 
VIp in Fig. 18(b) are 1:2:4:,-- and : 2^ \^r 
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If, therefore, similar circuits are used for other 
neuron output values or connection weights, the weigh- 
ing additions can be realized, as shown In Fig. 10(a). 
The block BLK2 shown In Fig. 18(c) can be used (as\ 1 1 ^ y 
is ^s\ the multiplier. The nonlinear transfer function ^ 
circuit has to be given the function of the AD con- 
verter for rewriting the arithmetic result output^edl v 



to the product/sum output line NO in the pluraK'roemory v/ 
cells in^^binary notations. For this*\necess I t^l, i t 
Is possible to use the embodiment shown in Fig. 19(a). 
In the embodiment of Fig. 19(a), a z (z = 2^) number 
of nonlinear transfer function circuits DSl, DS2 , - - 
,„ and DSz- and— an encoder are combi ned. These non- 
linear transfer function circuits DSl, DS2, - - and 
DSz are given the characteristics shown in Fig. 19(b) 
by adjusting the reference vo 1 tage VRx with the cir- 
cuit of Fig. 18(d). Then, the magnitude of the pro- 
duct sum of the neuron output values and the connec- 
tion weights can be known like the embodiment of Fig. 
18(0) from the number ^ig^f one^ of \th^ outputs NAl , NA2 , ^ 
- - and NAz ^^have*^^#^ high potential. Then, the J 
equivalent expressions of z bits have to be changed 
into binary expressions of p bits by the encoder \o 
that^'^lhey have to be transmitted to the write circuit ^ 
through the p number of output lines NVl , NV2 , - - 
and NVp. It follows that the encoder of Fig. 19(a) 
may be given" the Input/output relations^' shown in Fig. ^ 
19(c). This encode'r can be real i zed wl thout dif- 
ficulty. An example of the structure for p = 3 is 
shown in Fig. 19(d). The present embodiment can be 
easily extended to the cases other than that for p = 

The description ^hus fa^ made'^ls exemplified by n/ 
the multi-layered neural network. Despl te ^f^ th I s ex- y 
empl I f Icat Ion, however, the present Invention should 



not be limited to the mul t i - 1 ayered neural network but 
can be applied to other types of networks by using the 
embodiments thus far described. Figs. 20(a) and 20(b) 
and Figs. 21(a) and 21(b) show embodiments for realiz- 
5 ing the data processing using the Hopfleld network ac- 
cording to the algorithm of Fig. 5(b). Fig. 20(a) 
shows an embodiments'^ 1 n which the unsynchroni zed Hop- v/ 
field network is realized by using memory cells one by 
one for expressing the neuron output values and the 

10 connection weights. As has been described with refer- 
ence to Figs. 2 and 3, the basic arithmetic method Is 
commonly shared between the multi-layered network and 
the Hopfield Hopfleld network, how- 

ever, the "gxi thmet ic^"" are carried out by using the 

15 neuron output values from all the neurons including 
those of itself. In Fig. 20(a), therefore , al 1 t;^e 
neuron output values are stored in one word line of 
^he) array A. In array B, as shown, the connection v 

weights necessary for calculating one neuron output 

20 value are stored on a common word line. The updating 
of the neuron output values can be executed in the 
following manner. In order to update the neuron out- 
put value Vx, for example, the word line WA of ^e^ ar- >/ 
ray A and the word line of j = 1 of ^hi^ array B are ^ 

25 raised. As a result, the new neuron output value of 
g(TixVx + TxaVa + - - - + Tx«Vn) Is calculated. This 
value may be written in the memory cell which is lo- 
cated in the positibn of 1 = 1 on the word line WA of 
[the) array A. The \updat i ngs of th^ other neuron output sJ 

30 values ar^'^^'^^^l ml 1 ar^. The value for example. Is up- / 

dated by raising the word line WA of array A and ^ 

the word line of j = 4 of ^h^ array B. As a result, 
the new value of g(T4iVx + T^aVa + - - - + T-mVn) 
is calculated. This value may be written in the 

35 memory cell In the position of I = 4 on the word line 
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WA of (th^ array A. Thus, the ^ri thmeti cs^^of the un- t/ 
synchronized Hopfleld network can be executed by up- 
dating the neuron output values Vi in the desired or- 
der. The ^rithmetic^ of the synchronized Hopfield y 
5 network can be easily realized by using the memory 

cells on the word line WAl of ^e\ array A for storing 
the neuron output values at present and by using the 
memory cells on the word line WA2 for storing the new 
neuron output values, as shown in Fig* 20(b). First 
10 ^f all], the word line WAl of ^he\ array A and the word 

line of j = 1 of ^he^ array B are raised. As a result, v/ 
the new value Vx of gCTxiVi + TiaVa + - - - + TxnV„) 
is calculated. "This value may be wr i t ten i n the 
memory cell in the position of i = 1 on the word line 
15 WA2 of the array A. Subsequently, the neuron output 
values Va, V3, -» and Vn are updated and written 

in the memory cells on the word line WA2 of fth'^ array^ ^ 
A. When |he updatings of] all the neuron output values" 
^re ende^, th^'^*'^pdat i ng of the neuron output values) is ^ 
20 continued by interchanging the roles of ^e\ word lines 

WAl and WA2 of l^h^ array A such that ^ej word line WA2 v/ 
is selected for calculating the neuron output values 
(whereas th^'^word line WAl is selected for storing the \/ 
neuron output values. From now on, the process|lngs ^ 
25 are 1 ikewi sej proceeded] by interchanging the roles of ^ 
{t^^ word lines WAl and WA2 of [thj array A. Thus, rac-| \/ 
Tcofding to the embodiment of Fig. 20(b)rA/ the^{ar i th- ^ 
metic^^of the synchtonized Hopfield network can be ex- ^ 
ecute< 

30 Likewise, the Hopfield network can be realized by 

using a plurality of memory cells for expressing the 
neuron output values and the connection weights. Fig. 
21(a) shows an embodiment for realizing the unsyn- 
chronlzed Hopfield network by using Ith^ p and q num- ^ 

35 ber^^Jof memory cells equivalently for expressing the v/ 
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neuron output values and the connection weights. Like 
Fig. 20(a), all the neuron output values are stored in 
one word line of ^h^ array A. Here, ^he^ p number of ^ 
cells express one neuron output value, ^he storage of ^ 
thej^rray B I's^jmad^ such that the connection weights \/ 
necessary for calculating one neuron output value are 
arrayed on a common word line for every q number of 
cells. The ^ipda ting of th% neuron output values may 
be'^ig^xecuted llkS^ the embodiment of Fig. 20<a). Since, v/ 
however, ^he\ p number of memory cells areM ndi vidual ly v 
^sed^ f or expressing the neuron output values, ^] p num- ^ 
ber of output lines of the nonlinear transfer function 
Circuit D are provided so that the \ar l thmet I ^ resul ts v/ 
may be written in parallel in \th^ P number of cells. i/ 
The synchronized Hopfield network can also be easily 
realized like Fig. 21Cb) i f t wo word lines of ^^ar- ^ 
ray A are used ^ik^'^Fig. 20(b). Likewise, it is quite ^ 
natural that the synchronized and unsynchroni zed Hop- 
field networks can be realized by binary expressions 
using' p and q number^^of memory cells for expressing ^ 
the neuron output values and the connection weights, 
as shown in Fig. 10(a). 

Figs. 8(a) and Figs. 20(a) and 20(b), and Fig. 
g(a) and Figs. 21(a) and 21(b) present basically iden- 
tical structures. If, therefore, the embodiments of 
Figs. 11 to IQ'i the data processings according to the 



embodiments of Figs. 20(a) and 20(b) and Figs. 21(a) 
and 21(b) can be eas i 1 y real i zed . Incidentally, in 
the Hopfield network, the procedure of continuing the 
updating of the neuron output values falls in the so- 
called "local minimum", in which the energy is not the 
mi nimum^ but ^h^ mi nimal , so that the neuron output 
values are no^3*'^ar i ed^ny more]. In order to avoid 
this, the well-known "quas i -anneal I ng me thod" can be 
used. The method of changing the shape of the non- 
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linear transfer f unct i on gradual 1 y is known for / 
realizing the quas i -anneal i ng method, as5^ described on V 
PP, 122 of Neural Network Processing (published by 
Sangyo Tosho and edited by Hideki Asou) . According to 
5 the present invention, this method can be easily real- 
ized by switching a plurality of nonlinear transfer 
function circuits D having different characteristics 
and by controlling the characteris tj^cs of the non- 
linear transfer function circui ts 1b*jf rl)ra the outsidej.v/ 

10 Al though ^here have been described^ examples in ^ 

which the neuron output values and the connection 
weights have been handled as positive numbers mainly 
in the mul t i - 1 ayered or Hopf ield network*; i t may be v 
convenient depending upon the application thaf^both or v 

15 one of the two values can take positive and negative 
values. The present Invention can be easily appj'ied 

<c / 

to such'^case. Fig. 22 shows one embodiment of the V 
present Invention, In which both the neuron output 
values and the connection weights are enabled to take 

20 positive and negative values. In Fig. 22, the neuron 

output values are stored in ^h^ memory cell array A, ^ 
and the connection weights are stored in memory \/ 

cell array B. The individual values are expressed 
with p or q bl tsj I ndi cat i ng the absolute magnitudes^ 

25 and with 1 bit indicating the codes. The bits in- 
dicating the codes (as will be called the "coding 
bits") Indicate a positive value with "1" and a nega- 
tive value with "O**. Of the neuron output values and 
connection weights thus read out by the method similar 

30 to those descr ibed^'^erel n befor^, the portions of the ^ 
p or q bits Indicating the absolute values are In- 
put^ed^ to the adders al , - - and an and bl , - - ^ 
and bn so that the resultant analog values are in- 
putl^ed^ to the multipliers ml, - - and mn. Inciden-N/ 

35 tally, when the neuron output values and the connec- 
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tion weights are to be expressed In %h^ binary nota- ^ 
tion, the data of the individual p and q bits inputie^N/ 
to the aforementioned adders al , - - and an, and bl 

- - -, and bn may be weighed and input^d^ like Fig. ^ 
10. On the other hand, the coding bits are input^edj, ^ 
as shown in Fig. 22, to exclusive OR circuits EORl , - 

- -, and EORn. In^case-'the coding bits fail to become'/^ 
similar, namely, when the result of^mul tipl ication|] is / 
negative, the outputs of the aforementioned exclusive 

OR clrcui fe^lake th^ high Qevel. In case of (th^ sirai- v/ 
larity, namely, when the multiplication result is 
positive, the outputs of the exclusive OR circuits 

(take th"^ro w feve ij. " Swi tches "SWl , , and SWn ^ 

operate to transfer the outputs of the multipliers to 
adders cl , - - and cn, when the outputs of the 
exclusive OR ci rcui ts [take th^To'w ITeveJ, \and |be sam^ >/ 
^•Atd, the multipliers cl ' , - - -, and cn' when the out- ^ 
puts Of the exclusive OR circul ts ^ake th§'' high \Level\. ^ 
As a result, the sum of the positive results of multi- 
plications is output^ed] to the product/sum output line n/ 
NO, and the sum of the negative multiplication results 
is output^e^Ii to the product/sum output line NO*. In >/ 
the nonlinear transfer function circuit D, the dif- 
ference between the signals of the product/sum output 
line NO and the product/sum output line NO* is con- 
verted into a digital value of p bits and fed to the 
bus ABS so that the coding bits are determined accord- 
ing to the magnitudes of the signals of the product/ 
sum output line NO and the product/sura output line NO' 
and output^d^to a bus SIGN. Incidentally, it is 
Ceasi l^i'^posslblei'according to ^h^ method? si mi lar to the 
taforementionedj ones> to give the nonlinear characteris- V 
tics, as shown in Fig.. 9 or 10,^ccordln|i to the ex- ^ 
pressions of the neuron output values. According to 
the present embodiment, both the neuron output values 
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and the connection weights can take positive and 

negative values. As a result, the present embodiment , 
is advantageous in that the range for applying \the^ ^ 
data processing is extended. Although both the neuron 
output values and the connection weights are (e^able&l ^ 
to take ^the\ posi t i ve and negative values, it is easily ^ 
possible to make modi f i cat ions ^suchy'that either^^f ^ 
them] tak^^h^ pos i ti ve valuers]. ^ 

The description thus far made has been directed 
to the embodiment in which the product/sum function 
necessary for calculating the neuron output values and 
the nonlinear transfer function circuit are realized 
^^the arithmetic circuit. Despi te fcfl this descrip- 
tlon, however, a circuit for other ^r i thmet ic|''can be 
added to the arithmetic circuit. For example, the 
data processing system according to the present inven- 
tion can be applied to the so-called "classification 
problem" such as the speech recognition or the letter 
recognition, in which input patterns are classified 
into several classes. In this case, the comparator Is 
conveniently disposed in the arithmetic circuit, as \/ 
has been descr ibed^^herel nbef or^. In the classifica- 
tion problem, a desired value corresponding to a cer- 
tain class can be attained as the output l n'^ case"" the 
inputted^ patterns are apparently classified ^^''t*^® ^ 
class. In the delicate case, however. In which it is 
questionable to determine which ofplurar classes the \/ 
lnput^ed\ pattern belongs to, the classification may 
fall ^t middle, between the desired values of the V 
pluraKcl asses . I n'^ case'' the I nput\te<a speech is *K* in v/ 
the speech recognition, for example. It Is coded, and 
the connection weight Is ^oj set^^'that the neuron output ^y 
value (or the desired value) of 1111 may be obtained 
in the output layer tfo^'^the speech wavef orms*" ^l ven to) ^ 
the Input layer. If the Input Is 'C* , the connection 
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weight is (sol se t that the output value (or the desired y 
value) of 0000 may'^fce issued* Iff in this case. \^^x\:<. 
middle speech waveform between ^h^ *K' and^hj ^ 
the neuron output value of the output layer ma^1|,utputl v/ 
a middle value such as 0001 or 1110. In this case, 
the distance (or similarity) between the neuron output 
value of the output layer and the desired value of 
1111 for the 'K* or the desired value 0000 for the ♦C 
can be interpreted as measures |pr glvingj'^the similar- V 
ity [to the *K* or of the input speech. It is, 

therefore^ convenient to give a function to determine^] 
the distance between -the output result and the desired 

"^alue by providing the^ circuit with a cir- 

cuit for comparing the neuron output value of the out- 
put layer and the desired value of the class. 

Fig. 23 shows one embodiment^ i lij whi ch (ther^ are] 
i n tegrat in one semiconductor chip: an arithmetic ^ 
circuit 12a for comparing the neuron output values and 
the desired value; and an arithmetic circuit 12b for 
calculating the neuron output values. In Fig. 23: 
the desired value is stored in the memory TG; the neu- 
ron output values are stored in (th^ memory A; and the 
connection weights are stored in l^h^ memory B. The \^ 
calculations of the neuron output values may be !ac- 
compl ishedj by the method similar to those thus far ^ 
described, by reading ^utj the neuron output values 
from ^h^ memory A and the connection weights f rom ^he] ^ 
memory B, by calcul'ating the neuron output values with 
the arithmetic circuit 12b and by writing the calcu- 
lated result in [Ihe^ memory A. The comparisons are \/^ 
carried out by reading the neuron output values from 

(Ih^ memory A and the desired value from Q,he\ memory TG, \/ 
by determining the distances in parallel with the 
arithmetic circuit 12B, and by writing the result in ^ 

^h^ memory TG or ^utpu 1 1 i n^t^ie same through the \y 
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input/output device. Since, in the present embodi- 
ment, both Jthe] memories TG and A and the arithmetic 
circuit 12a are formed ^ve^^'^the common chip, the num- 
bex^'iof [thei buses 1 and 2 can [i>^ eas i ly^i ncreased to . 
5 process the numerous bits in parallel. This results 

injjtn advanta^*'' that the distances can be calcula^ted v/' 
at {a"Jhigh speed'. Incidentally, in the s true ture'' O^u^ 
far described, it is convenient to divide the arith- 
metlc mode .into'^ the neuron output value- calculating 

10 mode. ^o^''calculat\lji^ the neuron output values;, and^ the ^ 
comparison mode io^L^'^^compar^^^ the neuron output values 
and the desired valu^ to determine the distance.^ The/ 
switching of the ar i thmet i c modi^'" can be" v/ 
in response to the two arithmetic circuit control sig- 

15 nals NeT and NE2(J_for example^. Specifically: the 

memory mode may be selected if both the signals ^El 
and NE2 are ^t th^ highflevelj; the neuron output value 
calculating mode may be selected if the signal NEl is 
^t ^thej^ low (level\ whereas the signal NE2 Is^t the^high ^ 

20 \level\^; and the comparison mode may be selected if the \^ 
signal NEl is^t thej^high ^evel^whereas the signal NE2 ^ 
is^t Ih^ lowjlevel). Incidental ly, in the embodiment 
Of Fig- 23, the memory is divided into Ehrej'' whereas i/ 
the arithmetic circuit is divided into^llw^, but these 

25 divided circuits may naturally be mixed tove'^'the chip, \/ 
As has been described Lerei nbef orS, according to the 
present embodiment, it is possible to determine the i/" 
distances between ^he neuron output values and the 
desired value ^t a high spee^. As^^a result, the data 

30 processing speed can be acce 1 eratedf Ij^n ^.a^e ^ t^^^ls^ 

necessary, as in \the^pattern recognl t i on*^ Us i n^"" the ^ 
multi-layered networkQ"] to compare the neuron output 
values and each desired value to determine the dis- * 
tances \^ njbe tweeir; 

35 Fig. 24 showsyone embodiment of the arithmetic .-• 



circuit 12a SfJ FiS- 23, i.e., a circuit for comparing 
the neuron output values of the outputf^layer and the 
desired value to calculate the Ihummin^ distances in- v/ 
between- In the following, it is assumed that ^e^ 
5 memories TG and A of Fig. 23 are of the type, in which 
the data of the memory cells are read out to the data 
line pairs, as in Fig. 11, Fig. 16 or Fig. 18, and 
that the memories have ^ej arrays TG and A, respec- 
tively. The circuit of Fig. 24 is composed of a com- 

10 parator CMP and a comparison result converter COMFOUT. 
The comparator CMP is composed of a comparator CMPU 
and a load resistor Rcmi» connected in parallel, and 
the comparison result converter COMPOUT is composed of 
differential ampl i f iers AMP211 , AMP212, - - and 

15 AMP21Z. The comparator CMP is connected with the data 
lines DTGl and DTGl , - - -, and DTGr and DTGr of /the 
array TG, and the data lines DAI and DAI, - • -, and 
DAr and DA? of (Lh^ array A. Here, letter r designates y 
the number of memory cells on one word line and takes 

20 the value of n. if the neuron output values are ex- 
pressed in 1 bit, and the product of n and p if the 
neuron output values are expressed in p bits. Accord- 
ing to the present embodiment, it is possible to cal- 
culate the %ummi ngl^'^dis tances between the data read out v/ 

25 onto the data lines DTGl and DTGl , - - -, and DTGr and 
DTGr of %h^ array TG and the data read out onto \the^ \/ 
data lines DAI and DAI , - - and DAr and DAr of \thej v/ 
array A. The opera^'tions of the present embodiment 
will be describe<^\ln the following. Flrst\of al^, a 

30 clear signal * c is raised In advance to turn on a MOS 
transistor Q216 and to break the gate voltage of a MOS 
transistor Q215. After the clear signal * c has been 
broken so that the signal Is read out to the data line 
to set the data line potential to Vcc or 0 V, the com- 

35 parator is started by a comparator starting signal * 



CMP. Then, the ^ogic ofj exclusive OR is taken in each -v/ 
of the groups of the data lines (DTGl , DAI), (DTG2, 
DA2), - - and (DTGr, DAr) connected with the com- 
parator. As a result, '^he gate of the MOS transistor 
5 Q215 is left^^t th^ ^ovlj ^o ten t i a^^ i f?f'case^*1;1ie data 

are similar on the data 1 1 nes'* - the] array TG and the \^ 
data l ines ja^t th^ ai^rdyV, ^u§ o therwi se^lsT \trans i te^ ^ 
jto th^ hish ^otentla^- Then, the MOS transis^tor Q215 
is turned on in the comparator CMPU whlc^'^the data 
10 are not (simi la^beTween the data lines of ^h^ array TG ^ 
and data lines of (jh'J array A* As a result, ^h^ ^ 

lore current will flow from the power source VCMP 
through the load resistor RCMP "to the earthed elec- 
trode,\f or the larger number of groups of the data \ 
15 [lines (DTGl, DAI), (DTG2, DA2), - - and (DTGr , DAr ), \ 
Jn which t he data are not similar/gj As a result, ,the 
potential of the compare line CO will be ^he^ lower for^ 
^he^ larger number^ of groups in which ^h^ data are not 
similar. The compare line CoAis connected with the \y 
20 differenti al amplifiers AMP21l|, AMP212, - - an d_ 

AMP21^^^ jrrs posed in the comparison result converter^ ^ 
^^OMOU^^Jlf the reference voltages VRCl , VRC2 , - - 
and VRCZ of those differential amplifiers are set to 
suitable values, the number of those of th^ comparison n/ 
25 result output lines DCOl, DC02 , - - and DCOZ; which v 
\take thef^high potential], the 1 arger [f or the larger] l/ 
drop of the potential ^f^the compare line CO, In ^ 
other words, the comparison result converter COMPOUT 
operates as a kind of AD converter. Thus, according 
30 to the embodiment of Fig. 24. the data read out to the 
plural^^data lines o^ \th^ array TG and the data read c/ 
out to the plurar^data of^(thJ array A can be compared / 
to determine their Vummln^ distance. If, therefore, 
one word is selected from each ibf the arra&J TG and [the* ^/ 
35 array] A, the data stored in the memory cells on the 
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selected word lines can be compared with each other. 
If, therefore, the desired values are individually 
stored in the memory cells on ^he^ array TG, it can be n/ 
compared with the neuron output values^'] which are \/ 
5 stored in the memory cells on one word line of ^he]ar-\/ 
ray Ar,u''to know what desired value the neuron output V/ 
values are close to and how close they are. '^|'n*case'£ 
therefore^ the obtained neuron output values are not ^ 
similar to the desired value corresponding to the 
10 class, it is possible to know at Qt^^hlKh ^peed what v 
class the neuron output values are close to and how 

close they are. 

' rncidental ly , In the "embodiment of Fig. 24, the 

result output]le<^ to the comparts jni^^resjgl t output lines v/" 
15 may be output^ed^CP ^^e outside o|\^the chip through ^ 
the Input/output circuit at each time of comparison. 
Alternatively, the capacity of ^hei memory TG may be 
made larger than that necessary for storing the desir- 
ed values so that the results may be ^c*^ written' In 
^h§^ memory TG and then outputjted altogetherjr ^ 

Finally, an embodiment for. further speeding up 
the system of the present Invention by using a regis- 
ter viU be .described Vln the following. As has been •>/ 
described hereinbefore), according to the present In- 
25 ventlon, the neuron output values are calculated by 
reading out the necessary data from the memory, by 
determining the neuron output values with the arith- 
metic circuit, and 'by rewriting the determined result 
in the memory. In other words, one arithmetic mode 
30 (i.e., the neuron output value arithmetic mode) cycle 
is composed of the reading operation and the writing 
operation, and the arithmetic circuit Is Inactive In 
the writing operation. If* therefore, the time pe- 
riodHfor which the arithmetic circuit Is iiQactlv^"]^ 
35 is shortened, the" arl thmet ic mode can be further 
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Speeded upjV \ Flg> 25 \ shows one, embodiment in which the 
^*^^'''^ari thmetic mode Is l^peeded u|*^on the basis of the 

aforementioned point of view. The embodiment of Fig. 
25 is made by adding the register and the switches 
5 SWl, - - and SWr to the embodiment of Fig- 7. Ac- 
cording to the embodiment of Fig. 25, the neuron out- 
put values can be calculated at ^al high speed^ by using v/ 
the algorithm of Fig. 5, In the following, the de- 
scription will be made upon the multi-layered network, 

10 but similar effects can be attained even in the Hop- 
field network* In the embodiment of Fig. 25, the out- 
put value of the first neuron of the s-th layer is 

calculated "by "raising one word line of the memory cell 

array A to read out the neuron output value of the (s- 

15 l)th layer, by closing the switches SWl, - - -, and 
SWr to write the neuron output values of the (s-1') th 
layer in the register 14, and by opening the switches 
SWl, - - and SWr. Next, one word line of the memo- 
ry cell array B is raised to read the connection 

20 weight between! the neuron of the (s-l)th layer and the 
first neuron of the s-th layjr(^ andythe neuron output (/ 
values of the (s-l)th layer are read out by <yie]] regl s- v/' 
ter 14 so that the output value of the first neuron of 
the s-th layer is calculated by the arithmetic circuit 

25 12. The calculated results are wr It ten^l n fthe) memory \/ 

cell array A. ^mul taneously with thisJC" one word line ^ 
of ^e]memory cell array B Is raised to read out the 
connection weight b'etween the neurons of the (s-l)th 
layer and the second neuron of the s-th layer, and the 

30 neuron output values of the (s-l)th layer are read out 
by the register 14 so that the output value of the 
second neuron of the s-th layer by^he^ ^ 

arithmetic circuit 12.. ^rom now oQp the output values 
of the neurons of the s-th layer are likewise calcu- 

35 lated. Next, the output values of the neurons of the 
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(s+l)th layer are calculated by raising one word line 
of^hej memory cell array A to read out the previously)/" 
determined neuron output value of the s-th layer, and 
by closing the switches SWl , - - and SWr to write 
the neuron output values of the s-th layer in{thejre- V 
sister 14, and the subsequent calculations (arS pro- \/ 
ceedjed lik^ before. .As has been described herein- 
before, according to the present embodiment, the writ^ 
Ing operation and the reading operation can be simul- 
taneously carried*' bj^ providing Sh^ register 14 \so that 
they can be accomplished at a high speed^ 
'■""The description thus far made Is directed mainly 
to "the method of calcuTatlng" the neuron output values 
according to the present invention, and th^e^n^ecessary 
connection weights are assumed to ^ej al ready'' gi ven . 
Depending upon the subject, the necessary connection 
weights are easily given at the start or have to be 
determined by the so-called "learning". In the learn- 
ing for the multi-layered network called [th^ "back ^/ 
propagation", for example, several neuron output 
values (or test patterns) of the input layer can be 
prepared in advance to determine the connection- 
weights so that the desired neuron output values may 
be obtained in the output layer for the test patterns. 
As described in Section 2 of Neural Network Data Pro- 
cessing (published by Sangyo Tosho and edited by 
Hideki Asou) , moreover, ^ere is known] the learning \X 
algorithm for settling the connection weights such that 
the balanced state of the neuron output values ^^^^^^^^j,,^^ 
take the desired state even in the Hopfield network*. 
This learning can be applied to the present invention 
by the following three methods. According to the 
first method, the learning is carried out by using an 
external computer, and the obtained connection weights 
are written in the data processing system according to 
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the present invention* This method is advan tageous ]in 
tha^^'^the learning can be executed by the software so 
that the learning algorithm can be easily changed, but 
^is difficult to speed up the learning. According to 
the second method, the arithmetic circuit of the sys- 
tem according to the present invention is given an 
arithmetic function for the learning so that the 
learning is executed on-chip. This method speeds up 
the learning but may be difficult to integrate all the 
circuits necessary for the learning over a common 
chip. The third method is an intermediate one between 
the first and second methods, and a portion of the 
^i thmetic^ necessary for the learning is executed by ^ 
the system of the present ^ >^ven t i oa \£i«'eas the re- 
maining portion of the ^r i thmet ic^necessary for the. 
learning is executed by the external computer. fhls 
method is advantageous in that it can accelerate the 
learning speed more than the first method and that the 
arithmetic circuit of the system of the present inven- 
tion can be simply constructed. This third method 
will be specifically describe^^'frr the followin|. In- 
cidentally, the learning method is exemplified by the 
back propagation method in the mul t i - 1 ayered network. 
In the back propagation method (as will be shortly re- 
ferred to as the "BP" method), the connection weights 
are updated according to the following formulas: 

T^ij = T^ij + C djaVia-i (1); 

dj« = (tj - Vjm)g'(Uj«) (2); 

and 

djs =g'(Uj3)E t (T^^^ijdi3-i) • (3) 

(s = m-1 , - - - , and 2) , 
wherein: 

c : a small positive number; 
tj: a target of the neuron outijut value Vj^ 
of the final layer; 
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g': a derivative of the nonlinear transfer 
function si and 
Uj»: a quant i ty before passage; through the 

nonlinear transfer function circuit g in 
5 the j-th neuron of the s-th layer, as 

will be defined by the following 
formula: 

Ujs = E 1 (T«-^jiVis-x + e ja) (4). 

The connection weights may be updated by determining 
10 the quantities to be updated from the above-specified 
formulas (1) to (4) for every input data for the 
learning and by using the sum of the updated quanti- 
ties of all the input-data for the learning. On the 
other hand, the updating may be carried out by adding 
15 the following term called the "Inertia term" to the • 
formula (1): / 

fi A T«i J • - - (5), 

wherein: 

fi : a small positive constant; and 
20 AT®tj*: a corrected quantity of the previous 

updating. 

The updating Is continued £il^''the difference between ^ 
the neuron output values of the last layer and the ^ 
target values become^*} sufficient ly small. 

25 The learning Ithus f arj descr Ibed'^ can be executed ^ 

by the embodiment shown In Fl^. 23 and the external 
computer, as will be descr lbed^\|n the following. The v/ 
description to be dade Is directed to the case in 
which updating Is executed by summing the updated 

30 values of all the Input data, but s 1ml 1 ar operat ions 
are applied to the case in which the connection 
weights are updated for every Input data. Inciden- 
tally, the description to be made Is directed to the 
case of the three- 1 ayered ne twork , but similar opera- 

35 tlons are applied to the case of a netQw^^^ having. 
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three or more layers* 

First |of allj, the input data for all the learn- \/ 
ings and their target values are written in It h^ memo- \/ 
ries A and TG, respectively* Next, tL^eJ a random num- \/ 
5 ber having a small absolute value is written as the 

initial value of the connection weights I n Cthe] memory 
B. Moreover, the first input data are read out as the 
neuron output values of the first layer to the arith- 
metic circuit 12b, and the connection weights between 

10 the first and second layers are read out to the arith- 
metic circuit 12b by |t_he^ memory B, These values are v/ 
multiplied in parallel by the aforementioned method so 
that the neuron output values of the second layer are 
calculated and written in ^e\ memory A. Subsequently, \/ 

15 the neuron output values of the third layer are calcu- 
lated and written in [th^ memory A* The cal cul at i^'ons 
thus far described are executed for all the [leanin^'^ ^ 
input data to read out the neuron output values of the 
individual layers for the individual input data, the 

20 desired values for the individual input data, and the 

connection weights to the memory louts ide of the chip^v/' 
Next, the quantities for updating the connection 
weights are calculated in the external computer, and 
the updated connection weights are written In (lhe\ ^ 

25 memory B of the system according to the present inven- 
tion. Incidentally, the term g'(Ujs) appearing In^h^-/ 
formulas (2) and (3) may be either calculated from the 
value UjaMnput^e^ 'to the nonlinear transfer function \/ 
circuit D^^when the neuron output value Vjs is to be ^ 

30 calculated In the system of the present Invention, or 
calculated inversely from the value Vja by the exter- 
nal computer according to the following formula: 

g'(Uj3) = g*(g"^(Vjs)) - - - -r (6)* 

In order to add the inertia term of ^e^ formula (5), 

35 on the other hand, the corrected quantities of the 



connection weights may be stored in the memory outside 
of the chip for every updat i ng^"\ so that they may be ^ 
added to the newly determined corrected quantities in / 
accordance wi th ^e^ f ormul a (5), ^ 
5 The updatings thus far described can be repeated 

\o proceed\ the learning^. In order to know how the v 
learning advances, the distances between the neuron 
output values of the last layer for the individual in- 
put data and their desired values can be used as Ithi\ ^ 

10 measures. These distances can be calculated atGalhlgh ^ 
speed by using the embodiment of Fig. 24. As a re^^^^ 
suit, it is easily possible to confirm the (advanci of \/ 
learning while the learning is being accorapl ished.v/ 
As has been described hereinbefore, according to 

15 the present invention, the calculations of the neuron 
output values for the Input data for the learning can 
be executed atCa^hlgh speed In the system of the pre- ^ 
sent Invention. In the present invention, moreover, 
the memories composed of memory arrays are used in [Lhe^ ^/ 

20 memories TG, A ^nd B so that all the Input data, the 
desired values and the neuron output values of the 
preceding layer can be easily stored and so that the 
numerous bits can be read out in parallel by raising 
the word lines. As a result, the transfers of the 

25 data to the external memory can be executed altogether 
at^;]high speed. As a result, the learning can (b| v/, 
proceed^cTj at \a}hlgh speed. 

If the capacltfles of the memories are made suffi- 
ciently large in the present embodiment, the number of 

30 neurons can be easily changed according to the ap- 
plication. If, in this case, the neuron number is 
drastically changed, the nonlinear transfer function 
circuit may have to have its dynamic range changed, 
^or this necesSi ty;|; it is possible to switch and use a 

35 plurality of nonlinear transfer function circuits hav- 
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ing different characteristics and the reference volt- 
ages of the amplifiers Jji^ the non I i near trans fer func- 
tion circuit. In^cas^the neuron numbers are dif- 
ferent for the layers in the mul t i - 1 ayered network, 
the nonlinear transfer function circuit may have to 
have its dynamic range changed for the layers. This 
(necessi t^^fcan also be^oped^'^wlth by the similar y 
method. 

Incidentally, the description thus far made Is 
directed to the embodiment in which either the DRAM 
cell of the so-called "one transistor and one capa- 
city" type and the SRAM cells shown in Fig^s. 16(a) and 
16 (b ) are mlt i y "us ed , bu t o thernmemo ry ce 1 1 s can 
naturally be used in the present invention. Since the 
portion for storing the connection weights need not be 
frequently rewritten for \the!^ data processing, th^ v/ 
kinds of ^he^cells can be changed according to the 
contents of the memories by using non-volatile memory 
cells or the DRAM cells or the SRAM cells in the por- 
tion for storing the neuron output values. 

If the memory cell circuit is highly integrated 
by using very small memory cells such as Qh^DRAM y 
cells of the one transistor and one capacitor, some y 
memory cells may ^e\ some t imes'j^nac 1 1 ve because the 
wiring lines used are very small. The neural network 
is advantageous in that its function Is hardly In- 
fluenced even If the connection weights are changed 
more or less, but the data processing may be troubled 
in^case'^the memory cells for storing the neuron output 
values are inactive. In order to solve this problem, 
the redundant word lines or data lines to be used in 
the ordinary highly-integrated semiconductor memory 
can be provided so that defective cells may not be 
^sed. 

In Figs- 14(a), 14(e) indl 14(f) and Fig. 18(d), 
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moreover, (there are used the\ bipolar transistors'", 
which can be realized by the CMOS. Still moreover, 
the present invention can be practiced by not only the 
bipolar transistors and the MOS transistors but also'^ ^ 
other devices. 

Although the foregoing description is directed 
mainly to the mul t i - 1 ayered and Hopfield networks, the 
present invention should not be limited thereto but 
can be applied to^he^neural network data processing \/ 
f or networks of various types. For example, it is v/" 

possible to realize the network in which (The updating 
of^the neuron output values^such as'^the Boltzman's i/ 
"machThe. As described on" pp 27" of Neural Network 



Data Processing (published by Sangyo Tosho and edited 
15 by Hideki Asouft^^^e Boltzman's machine is featured, 
although the network shape is similar to that of /the 
Hopfield network^} in that the neuron output value (0 
or 1) is not uniquely determined by another product 
sum of the neuron output values input(ied^ to the neu- v/ 
20 rons and the connection weights* but in a probable man- v/ 

-tti y 

net. The probability P for the neuron output value'^ \/ 
take the value 1 is expressed by P = 1/ (1 + exp(-l/ 
T)). Here, letter I designates the product sum of the 
nol lnput^ed\ to the neurons and the connection wei- v/ 

25 ghts, and letter T designates a parameter called the 
temperature. The Boltzman*s machine described above 
can be easily realized according to the present inven- 
tion. For example^ the reference voltage VRx of the 
nonlinear transfer function ci^rcui t D, as shown in 

30 Fig. 18(d). is not set to [th'S^'^s teady value but may be \/ 
changed with time within the fluctuating range of the 
product/sum output line NO. Then, the neuron output 
values can be determined according to the probability. 
The effects obtained by changing the changing rate are 

35 similar to those obtained by changing the temperature 
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T. 

If the capacity of the memories is sufficient, as 

is apparent from the comparisons between Fig. 8(a) and 

s 

Figs. 20(a) and 20(b), various types of networK" can be 



5 realized by a common system merely by changing the ad- 
dresses of the memory cells for storing the neuron 
output values and the connection weights. Thus, the 
present invention has a [hiighl^'^wlde applicability. ^ 
Although the description Jhus far made is dl- 

10 rected to the app 1 i cat i ons ^^"Lh^ neural network data ^ 
processing, the present invention should not be limit- 
ed thereto but can naturally realize such a system *ln] 
'^'^^ high degree of integration as^is used for^lhe^data ^/ 
processing by connecting a number of processing ele- 

15 ments having similar process i ng funct i ons in the form 
of a network. / 

In the embodiments thus far described, the de- 
scription is directed mainly to the s true t.ure . i n which 
the arithmetic circuit performs the anal og^^r i thme- ^ 

20 tics}. The analog arithmetic circuit is advantageous ^ 
in that it has^'^high speed and a small circuit scale. v/ 
Despi te ^fj this description, however, the present in-v/ 
vention should not be limited thereto but can be used 
in a digital arithmetic circuit without departing from 

25 the gist thereof. In this case, the calculations can 

be \highly accuratel^Q execute'd'^ by the digital arith- v 
metic ci rcui t . 

As has been described hereinbefore, according to 
the present invention, the system for executing (the^ ^ 

30 data processing by combini ng (Ih^ memories and ^e\ \/ 
arithmetic circuits and by performing ^h^ parallel 
^ri thme t i cjfxwi th the arithmetic circuits like the v/ 
parallel distributed processing system such as the 



35 



neural network. In which a number^of ^ri^hmetic units 
for relatively simple ^rithmetic"^ are connected i n yth^ v 
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network form, can be realized with a high degree of / 
integration without sacrificing ^he^ speed, ^ 
It is further understood by those in the art that 
the foregoing description is^referred embodiment of \/ 
the disclosed device and that various changes and 
modifications may be made In the Invention without 
departing from the spirit and scope thereof. 
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ABSTRACT OF THE DISCLOSURE 

Herein disclosed is a data processing system hav- 
ing a memory packaged therein for real i zi ng (jT^l arge- x/^ 
scale and high-speed parallel distributed processing 
5 and, especially, a data processing system for iLhe^ neu- 
ral network processing* The neural network proces- 
sing system according to the present invention com- 
prises: a memory circuit for storing neuron output 
values, connection weights, the desired values of out- 

10 puts, and data necessary for learning; an input/output 
circuit for writing or reading data in or out of said 
memory circuit; a processing circuit for performing a 
proces^nV^f or determining the neuron outputs such as 
the product, sum and nonlinear conversion of the data 

15 stored in said memory circuit, a comparison of the 

output value and its desired value, and a processing 
necessary for learning; and a control circuit for con- 
trolling the operations of said memory circuit, said 
input/output circuit and said processing circuit. The 

20 processing circuit is constructed to include at least 
one ^ a'^ adder, a multiplier, a nonlinear transfer 
function circuit and a comparator so that at least a 
portion of the processing necessary for determining 
>he nexy^ron output values such as the product or sum c-.^-^ 

25 may be accomplished in parallel. Moreover, these cir- 
cuits are shared among a plurality of neujtjrons and are CX^ 
operated in a time sharing manner to determine the 
pluraT neuron output values. Still moreover, the 
aforementioned comparator compares the neuron output 

30 value determined and the desired value of the output 
i n paral lei. 



